Potentially dangerous sign_msg function should be removed

The potentially dangerous sign_msg function introduced in #82 should be removed. It is based on Ethereum's personal_sign, the use of which is discouraged since the introduction of the much superior EIP712 standard. EIP712 addresses the many issues and shortcomings of personal_sign. If the function is adopted in Stacks wallets, it will put the entire ecosystem on a perilous path. We should learn from the Ethereum space and skip this function completely.
personal_sign had two main use cases:

Signing messages to authenticate with some app/dapp.
Signing messages that authorise some on-chain action.

When it comes to the first, we already have an authentication scheme. Right now it does not allow us to prove address ownership, but that can be achieved in different ways. (I will come back to this at the end.)
The second one is more important, so I will focus on that. Signed messages have been used for many different things in the Ethereum space. I will take on one such example for brevity: off-chain order signing. 0xProtocol allows users to sign an order, which can later be submitted to a smart contract to make a token trade happen. Early versions of the protocol would hash an order structure and request users to sign the order hash via personal_sign to authorise the trade.
It first looked like this:

And then this:

Does the user have any idea what he or she is actually signing? Obviously not. The user will just have to trust that the dapp fed the right message to the wallet. The messages are also not domain specific, which means that a malicious app could trick a user into signing something meant for another platform. And since the messages that are passed to the wallet are hashes, there is no way to reconstruct the original message in the wallet. It is just all-round terrible UX and highly dangerous. In hindsight it is obvious that the function should never have existed, but back then it is all we had and such signed messages were cutting edge. They allowed for many new an exciting things. Although personal_sign was largely left behind, it is still causing longer-lasting damage because it normalised the concept of signing byte strings. Many people are now so used to signing on-chain actions in this way, that they just hit Confirm when that wallet screen pops up. The sign_msg function opens us up to all of it.
Signing byte strings should not be normalised in the Stacks ecosystem.
I cannot stress that enough. We have so many tools at our disposal to make the UX a lot better and safer. Ethereum did not have the benefit of a human-readable interpreted language like Clarity. There is no good reason to introduce a personal_sign equivalent for Stacks.
Besides, it seems to be carelessly implemented and actually commits the same mistake as the original personal_sign implementation.
https://github.com/Zondax/ledger-blockstack/blob/abaf1d01a0bef27b2a603ec3d15bb6a20ddbcc96/js/src/index.ts#L245

The \x19 is a (Bitcoin) varint representing the byte length of the prefix, which is correct for Ethereum Signed Message:\n as it's 25, but not for Stacks Signed Message:\n. It seems like it was copy and pasted thoughtlessly. It should have been \x18.
The message length is appended as an ASCII string, so you have no idea where the actual message starts. They made the same mistake in geth. https://github.com/ethereum/go-ethereum/issues/14794

The language in #76 almost makes it sound like sign_msg is a stopgap solution until we have something better. I really do not think that is the right approach. I rather have nothing, over this function, until a good message signing standard arrives. Hiro, as a shepherd of the Stacks ecosystem, should consider very carefully whether adding such a function is worth it.
There is a draft SIP that describes a structured data signing method much like EIP712. It describes a general structured message signing standard that leverages Stacks wire format, while still allowing safe human-readable text message signing. (If people insist.) I would welcome such a standard or one like it. And to go back to proving address ownership, the linked SIP would make that effortless as well.
I hope that all this is convincing enough to remove sign_msg. For reference, we are building a hardware wallet at Ryder and will not implement an unstructured message signing function such as this one for the reasons laid out above.

thanks for this clear explanation. in our case, because of the limitation the device has in regards to the characters that can be shown on the display, the ledger app only processes ASCII string messages,  So users would be able to see the message they are signing. .. trying to sign a message like the one you described would cause an error.

Thanks for reporting @MarvinJanssen.
If you can help me understand the varint prefix, Stacks Signed Message:\n is 24 chars, so wouldn't that make the prefix \x18 in little endian encoding?
You describe off-chain order signing as an example of how message signing is used, but as we follow the message prefix scheme, doesn't that mute the ability to build features like this and thus prevent any similarly related attacks? What would be an abuse vector of the sign_msg function as is, or are you more concerned about the practice of signing raw input messages in general?

If you can help me understand the varint prefix, Stacks Signed Message:\n is 24 chars, so wouldn't that make the prefix \x18 in little endian encoding?

actually, the variant prefix is \x19

@neithanmo I see. That makes things better. In that case I suggest the JS lib to enforce this and not have it sent to the Ledger at all if there are non-printable characters present in the message. I did not dive into the Ledger code and made the report solely on the linked PR so my apologies for not going deeper.
I think the length as ASCII should still be fixed.
To answer your question, I was unaware that the message signing function in the firmware rejects non-ASCII messages. personal_sign was at some point in the past (ab)used to sign hashes of data structures. The ASCII prefix had nothing to do with it. They did it by hashing an ABI encoded struct and requesting a signature from the user. So the final message would look like: \x19Ethereum Signed Message:\n32<32 bytes here>. Then on the contract they would reconstruct that message to verify the signature.
Here is one such example (that I even made myself):
https://github.com/wyvernprotocol/wyvern-v3/blob/b20ae7873d3f0e9b8f5ea6d0ff3947646830dbb7/contracts/exchange/ExchangeCore.sol#L196-L201
https://github.com/wyvernprotocol/wyvern-v3/blob/b20ae7873d3f0e9b8f5ea6d0ff3947646830dbb7/contracts/exchange/ExchangeCore.sol#L25
My concern with it is that users cannot verify what they are signing. But now that I know that the Ledger app won't accept non-ASCII messages, this is less of an issue as developers will be hard pressed to (ab)use msg_sign in the same way as Eth devs did with personal_sign.
Having said all that, I still wonder what the usefulness of msg_sign is. What are the use-cases? Even more so now I heard that Hiro is working on a SIP018(-like) implementation for Hiro Wallet. You could use SIP018 to sign a string-ascii CV if you want a text message.

@neithanmo I see. That makes things better. In that case I suggest the JS lib to enforce this and not have it sent to the Ledger at all if there are non-printable characters present in the message. I did not dive into the Ledger code and made the report solely on the linked PR so my apologies for not going deeper.

we can add this double check as well.

I think the length as ASCII should still be fixed. If you make it a (var)int then you can easily determine where the actual message starts.
https://github.com/Zondax/ledger-blockstack/blob/a7a0caa8434cfda23bc54b82555db7104e0c0a3a/app/rust/src/parser/message.rs#L94
Counting digits does not work if the user's inputs a message like "00", you end up with \x19Stacks Signed Message:\n200.

@MarvinJanssen  we considered that case as it is clearly described in the EIP-712 documentation as a possible flaw, that is why the parser verifies if  \x19Stacks Signed Message:\n200 correspond to a str.len = 200 or a str = "00" with length=2. for this message the device will show the user the string "00" and will ask the user if they want to sign it.
the case for strings starting with numbers is also considered, and last but not least, trailing zeros are also invalid.
there is one important thing. because of the small screen in ledger devices, and considering that string_messages can be huge. we limited the number of characters the device is going to show on the screen, which means that depending on the size of the screen, the message users see on the screen might not be a full message.

@MarvinJanssen we considered that case as it is clearly described in the EIP-712 documentation as a possible flaw, that is why the parser verifies if \x19Stacks Signed Message:\n200 correspond to a str.len = 200 or a str = "00" with length=2. for this message the device will show the user the string "00" and will ask the user if they want to sign it.

You have the opportunity to fix it, so why not fix it? There is no reason for parity with Ethereum's personal_sign.

there is one important thing. because of the small screen in ledger devices, and considering that string_messages can be huge. we limited the number of characters the device is going to show on the screen, which means that depending on the size of the screen and the message length, the message users see might not be a full message.

That sounds a little scary.  Perhaps it is better if the user has to explicitly enable it, like with "blind signing" for the Ethereum Ledger app. What is the limit of how many pages of text you can show?
As an aside, what kind of huge string messages are you expecting?

You have the opportunity to fix it, so why not fix it? There is no reason for parity with Ethereum's personal_sign.

we have already fixed that, it is how the parser was implemented.

That sounds a little scary. Perhaps it is better if the user has to explicitly enable it, like with "blind signing" for the Ethereum Ledger app. What is the limit of how many pages of text you can show?

we can show the complete message but it would be annoying the number of times users have to click..for any message we are going to show only the first 60 characters (header not included).
What we could do for cases where strings.len > 60 is to process them only if expert mode was configured on the device. but what is a proper limit here ?

As an aside, what kind of huge string messages are you expecting?

not sure, fixing a number maybe just too much or too few for some users

What we could do for cases where strings.len > 60 is to process them only if expert mode was configured on the device. but what is a proper limit here ?

Good question. I am not sure what the feature is meant for so maybe @kyranjamie can shed light on a good limit.

@MarvinJanssen @neithanmo https://github.com/hirosystems/stacks-wallet-web/issues/1051
I believe this issue prompted the ask of this feature.

i opened another issue to restrict signing messages that are longer than X bytes. If the app is in expert mode, such messages can be signed as long as they are valid. users know what they are doing so, the restriction of showing only the first X characters could remain active in both modes(I do not see an expert doing a lot of clicks)?.
To summarize:

in normal mode, any message longer than X would fail
if in expert mode longer messages are allowed.
in both modes, only the first X bytes are shown.

for now X = 60. the election of this number considered the nanoS screen, so a message with this length would require around ~3 clicks.

If our expectation is that most messages will be less than a certain value X (e.g. 60 characters), and that their length won't require users to "click too much" (e.g. 4+ clicks), then any cap we put into place for that will be irrelevant for most users.
For the few users that do encounter longer messages and more clicks, my sense is that it's better to show the full message and require more clicks, without requiring the extra step of enabling "expert mode" (and without limiting the display to only the first X characters). That way there is no possibility of "blind signing" even part of the message.
Of course, some subset of these few users may encounter many clicks (10+?) if they're signing something particular complex. But presumably that increased complexity will correlate with one's increased desire to verify the complete message and increased tolerance for clicks?

Although some upper bound surely is necessary still, lest dapps may try to send very large strings to be signed.

@kyranjamie @markmhx is there anything pending here?
should this result in some actions/tasks?

we agreed on showing the complete message to users..
please reopen if it makes sense or to start a discussion in regards to an alternative approach.


Adds support for Wagtail 2.0 and Django 2.0

Tests are currently passing against the beta release with both Django 1.11 and 2.0.
I don't envisage there being any changes between now and the official release, but I'd like to wait and be sure before merging.
It's unknown at this time whether wagtail-condensedinlinepanel will be updated to support 2.0 any time soon, so I might add a note about that to the installation instructions where it is recommended.

👍
I think the readme needs to be updated too. Very needed to have django 2/wagtail 2 support!


[WIP] Property tests for dependencies file parsing

As per #802 the dependencies file caches all lines for later serializing, so this round-test may not be that useful, however it can surely serve as a safety net for future refactorings.
The test doesn't yet generate all valid dependencies files, but it could already find some pitfalls:

can have comment line starting with #,
but can't have a hash-comment inline such as nuget FsCheck #fscheck is cool
can not have comments after: github, gist, http dependencies

Didn't fix those for now, instead made a workaround in generating test data

ready to merge?

Not yet, I'll do some improvements soon and let know when its ready

OK  I think it can be merged for now - added a bit more possibilities to the generator, however still there are some "hardcoded" values.
Can be a good start when time comes for refactoring DependenciesFile parsing

thx


I can change the main themes but the background color is still white

how can i fix it

This theme only affects the prompt. Background color depends on your terminal.

oh i see thank u.


Replace prepareStatement with "With" feature, create real prepared statements

Hi,
when using postgresql-async, I've noticed that we're actually "wasting" prepared statements, because they are implemented in a way that they can only be used in a one-time fashion.
Imho, the reason people are using prepared statements is to get some type safety and prevent SQL Injection attacks.
I was thinking, that it might be a better way to allow the simple binding / prevent sql injection using "WITH" queries, e.g.
WITH var(lastname) AS (SELECT 'Troy'::varchar(16)) SELECT * FROM users, var v WHERE lastname = v.lastname;
( see https://www.postgresql.org/docs/current/static/queries-with.html )
we could have something like
connection.createQueryWith("SELECT * FROM users WHERE lastname = $1", Seq("Troy"))
and a
connection.createPreparedStatement("insert into users values (?)") {
 stmt => {
 stmt.execute(Seq("troy"))
 stmt.execute(Seq("steve"))
 stmt.executeAll(Seq( Seq("chris"), Seq("michael"))
}
}

What do you think?

Is there any interest in this? I could try to create a PR, but probably would need some guidance.
I think I would have to create a own subclass of ClientMessage with appropriate Encoders that would prepend the WITH (...) part of the query ?

I don't understand this, where did you see that the client is not using actual prepared statements and is wasting them?

The problem is not "not using prepared statements" - the problem is, we're using prepared statements in a "just one query" fashion. Prepared Statements per se are made for multiple executions, yet the API provided allows only to execute them once.
The "WITH" statement would more or less give the same features of Prepared Statements with a reduced cost if they are only used in a one-time fashion (because they don't allocate any persistent data structure on the server).


Captcha Support

Does this script support recaptcha version 2?

Probably not. Hopefully some time in the next few months I'll add support for returning Promises in custom validators which will allow you to integrate with any async behavior.

Here's a complete example of reCAPTCHA working with this plugin.
http://jsbin.com/kirevujexa/edit?html,js,output


replace ttys?

Tests appear to be failing because of the issue described here. Given that the issue has been open for a couple of years, we should probably just find a different solution or fork it.

no longer used


Replace INC QAT and pruning example to dynamic quantization

Replace INC QAT and pruning example to dynamic quantization in quickstart.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.


[Feature Request] Support one click deploy on Render

Support one click deploy on Render  https://render.com/

@ahaapple i can do it, assign me.

Assigned, thank you


appliance/mongodb: Scale up on first resource request

Similar to MariaDB this has the MongoDB appliance only start up once the first resource is requested to be created.

LGTM pending package rename.

👍


yubico-piv-tool: add meta.pkgConfigModules, passthru.tests.pkg-config

Description of changes

Add meta.pkgConfigModules and passthru.tests.pkg-config
Use lib.cmakeBool, lib.cmakeFeature
Move check to nativeCheckInputs
Remove with lib;
Use --replace-fail in substitute

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[x] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

@ofborg eval


Draft: Associate label sets with wall profiles

Adds ability to associate custom label sets to wall samples. This can then be used by dd-trace-js to associate span ids with said samples.
Implementation notes
The implementation adds JS API to set and clear labels, as well as API to query whether labels were used when capturing a sample. In practice, the WallProfiler JS class will gain getters and setters for properties labels, a method named unsetLabels and a getter for read-only property labelsCaptured.
Internally, the implementation uses a PPROF signal handler executing before V8's signal handler used for profiling. It will associate the current label set with a timestamp, and put it in a preallocated ring buffer (so allocations are avoided in the signal handler.) This happens once every 10ms while profiling. The ring buffer is sized generously, for 2x the number of samples for a single profile. This allows capturing the next profile's labels while the previous profile is post-processed.
When the profile is post-processed once every minute, the labels are matched to samples based on the timestamp, and emitted into the serialized profile.
Timestamp matching
Timestamp translation is necessary between V8's timestamps and libuv's high-resolution timer. The timestamps don't need to be exactly matched (and most likely won't be.) V8 uses microsecond timestamps, libuv uses nanosecond timestamps, and the bases are different. After matching the bases and scaling, the labels with closest timestamp in the ring buffer within a ±5ms range of a sample are associated with the sample.

Superseded by #105


Search Button Click, how to track?

Hi,
I see delegates for history clicked, category clicked and listview clicked.
But don't see a way to track search button clicked?
Use case:
My search string does not match with anything in database. But I want to get the search string in the program once the search button is clicked.

anyone?


Fix Fix test_partial_update_schema_change

Proposed changes
Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

please add the description for this PR


Memory usage reported in Rancher UI is totally off

Rancher versions:
rancher/server or rancher/rancher: 2.0.0
rancher/agent or rancher/rancher-agent: 2.0.0
Infrastructure Stack versions:
kubernetes (if applicable): v1.10.1-rancher1
Docker version: (docker version,docker info preferred)
18.03.1-ce
Operating system and kernel: (cat /etc/os-release, uname -r preferred)
Ubuntu 16.04.4 @ 4.4.0-87-generic
Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
ESX
Setup details: (single node rancher vs. HA rancher, internal DB vs. external DB)
single node
Environment Template: (Cattle/Kubernetes/Swarm/Mesos)
rke custom nodes
Steps to Reproduce:

Login in to Rancher
View memory usage of the cluster, or that of single nodes in nodes tab
compare to prometheus stats, docker stats output, top, vmstat, etc on nodes

Results:
Rancher shows:

Data from Prometheus:

The prometheus data matches the one reported from "docker stats" on the node, e.g. for that gitlab instance:
CONTAINER ID        NAME                                                                                CPU %               MEM USAGE / LIMIT     MEM %
3dd3ede638bb        k8s_gitlab_gitlab-6dcd58f588-7lxr6_gitlab-external_5ab3e7bc-52f5-11e8-bf25-005056a8606f_0   3.76%               10.37GiB / 54.87GiB   18.89%

It seems Rancher shows the total requested memory (like "kubectl describe nodes"), not the actual node memory usage.
This might be pretty misleading if most pods run unlimited.

Workloads in general don't automatically reserve memory, which is why you probably don't see it going up.  But I just verified that this is working how it's supposed to.  When creating a workload, you can force memory reservation under Advanced options - Security & Host Config section.

Please consider improving this. It is pretty much useless now. You simply cannot see the real state of the cluster. It also counts NoSchedule master nodes and does not reflect the real usage.

I've seen nodes go (b)OOM because workloads got scheduled to them which they actually couldn't handle. I'm not sure if this is intended behaviour, but in the end it made me monitor pods more closely and define proper memory resource requests for nearly all of them. That's perfectly fine, and the cluster runs pretty stable now.
Just some rancher-internal things don't do that, like fluentd which seems quite memory hungry for a log shipper, and cattle-node-agent which seem to leak memory in an almost linear fashion.


Periodic jobs

It appears that there's a lot of interest in periodic jobs, and i think native support in Minion would make sense. Many existing implementations appear to be using a single server, often backed by cron, to schedule their periodic jobs. It has been suggested that cron+unique jobs (as proposed in #22) would be good enough, but i'm not so sure about that. What do you think, can we come up with something better?

Only dealing with unique jobs internally and letting cron handle the scheduling, does have the advantage that we don't have to care about testing.

I am extremely interested in not having scheduled / periodic jobs in cron.  One great use case might be a website that offers a service which needs to do routine actions on behalf of its users and therefore requires cron jobs.  But do you really want your website creating cron jobs?  Sure, it can work, but it would work and feel so much better being all integrated as part of the framework.
How does cron work?  Does it check for jobs every minute even if there are no jobs?  Minion, with it's Mojo::IO::Loop backend, could do the same, checking every minute to see what's scheduled and then run the necessary jobs.  There would of course have to be a DSL that allows for definition of the desired schedule and frequency, but once that's defined and understood how to parse, everything else is just running the tasks at the specified time.
Anyway, I'm totally in favor!  Would love to help but doubt that I can...  :(

I built a Job Scheduler at work that uses croniter, postgresql, and gearman.
The job scheduler scans the database jobs schedule (4000 schedules now which isn't ideal) able looking for things that need to be scheduled.
For each scheduled job we create a unique job id that can only be re-used for scheduled jobs. Since we wanted the ability to re-try a configurable amount of times. We have a job status table which (uses the unique job id as a fk) we use to see how many times a job has been re-run. We also create a unique job id for adhoc jobs, but they will only have a few statuses in our statuses table vs scheduled jobs that fail may have a lot more compared to adhoc if they are failing.
We will never have duplicate calls added to the queue since gearman supports that out of the box:
Gearman performs coalescence on the work sent by a client. If two or more clients ask for work to be completed on the same body of work, either by seeing that the same blocks are being sent or by using the unique value sent by the client, it will coalesce the work so that only one worker is used. It does this specifically to avoid thundering herd problems which are common to cache hit failures. from https://en.wikipedia.org/wiki/Gearman
Not sure if this will help at all. But I thought I would share some of my experiences.

I have a friend who would love to get away from using cron for scheduling. They are trying to build a billing system component for recurring transactions. As it stands, the do a big fat DB query via cron every minute/hour/day to see what might be due and then run it. I'm not sure if this is the right use case for him, but am intrigued by the possibility of native minion without cron.

Minion being a distributed system makes this a really hard problem to solve. You don't want all workers in the cluster try to reschedule jobs at the same time. So you basically need an election process, where one worker becomes the scheduler. And the election process needs to be repeated when the elected worker disappears from the cluster again. This would be a lot of work to get right.

The periodic job feature would be great. => +1
But as others pointed out I would vote for not relying on a system cron for that. Mainly because of portability (e.g. what about windows?).
Also reusing cron-like syntax (not another dsl please) would be great, since it's widely known and most people are comfortable with it

I currently use cron with curl to hit my app through https, where the controller of the matched route then enqueues the minion job. Cron is currently configurable to me and this works well and is pretty simple. However, is cron available on many PaaS and SaaS hosting platforms which promote developer friendly setups (cron may be viewed as not a part of a stack and a sysadmin tool)? If not, I would like to see the scheduling within Minion for the developer to control through the app.

The cron + unique job issues need to be separated out since I feel they're separate problems:

On Using cron: Cron is limited to Unix-like systems and also is limited to a minute level resolution. This can be severely limiting if your jobs need only say 5-10 seconds to run and you're waiting for the minute to complete before you can run the next set of jobs. This may or will lead to the system being idle at times or if too many jobs are scheduled for the same minute, then overloaded at the same time. I am not sure whether rolling a new daemon is the right solution, but it should be easy enough with AnyEvent for prototyping. This can allow jobs to run on wall-clock time and also on their own on arbitrary periods.

On Unique Jobs: What is defined as a unique job ? How does Minion know whether a job is unique or not ? Isn't this context/application specific ? If a particular job has to be unique, in my opinion, the user should make sure that the job is consistent with being unique by using the database backend. For example, if the user schedules a job twice and it needs to run only once, whichever scheduled job runs first, can set a flag in the db stating it is running and the other job doesn't run. The onus should be on the job to check this flag before running.

I guess it might be worth mentioning #7, which first brought up the unique job topic and describes a possible implementation best.

I'm really interested in how other popular job queues have solved this problem, so if you see a solution, please post a link! I'll start with Sidekiq, which appears to support periodic jobs in its commercial version. https://github.com/mperham/sidekiq/wiki/Ent-Periodic-Jobs

SLURM is a popular scheduler in the High Performance Computing space and tools like Sidekiq fall short of Slurm's overall capabilities like checkpointing, binding CPUs to jobs, etc.. Slurm's srun command supports scheduling jobs at fixed times with seconds level granularity unlike cron which is minute level.

Celery uses a separate (centralized) daemon to schedule periodic jobs. http://docs.celeryproject.org/en/master/userguide/periodic-tasks.html

Resque does the same. https://github.com/resque/resque-scheduler

Getting the feeling that maybe the job scheduler shouldn't be part of Minion itself, but a separate CPAN module.

I've not actually seen this anywhere yet, but i think periodic jobs could also work like retries. The minion_jobs table gets a new column recurring, which signals that the job should be repeated and contains information on when that job is supposed to run again. And whenever a job with recurring value transitions to the state finished, a new job with the same attributes gets enqueued automatically.
$minion->enqueue(some_hourly_job => [@args] => {recurring => '0 * * * * *'});

This would have a lot of advantages, exceptions and retries just work, these jobs would have the same lifecyle as every other job. But we would also need a way to stop these recurring jobs again, since the data persists in the database, they have to be stopped manually.

Another problem with this approach might be that there's a small window after the previous job transitioned to finished, during which the worker can crash and prevent the followup job from being enqueued. For retries this is not as important, since those jobs would just stay failed, but here the previous job is finished and can therefore get lost easily.

There are solutions though, we could just not let jobs with a recurring value transition to finished, and reset them to inactive directly. This would also help with removing recurring jobs again actually. Because now there could be a sync process, where each worker has a list of recurring jobs during startup and ensures that those are the only recurring jobs in the queue.
$minion->schedule(some_hourly_job    => '0 * * * *' => [@args] => {retries => 3});
$minion->schedule(another_hourly_job => '0 * * * *' => [@args]);

I might see a problems with the solution using new recurring column:

If the minion worker crashing, that the new periodic jobs would be enqueueing after minion_after seconds only, not on the schedule.
If the job takes longer than scheduling period, that new job would be enqueueing in invalid time. For example, job needs to be enqueued every minute, but sometimes it performed more than one minute.

@avkhozov I don't think there are any existing solutions with a higher level of accuracy.

I like solution with election a master worker. For example, every heartbeat-interval minion backend will elect new master worker (in register_worker method) through some algorithm and store flag is_master in backend database. While enqueuing the new periodic job from worker we can pass new master option to enqueue method and job will be enqueued only if master id in options equal current master in backend database. So all minion workers will be do $minion->enqueue(task => [@args] => {%$options, master => $workek->id}) by schedule, but only one job will be really enqueued. And while we have at least one minion worker in cluster we can properly enqueue periodic jobs.

https://github.com/kraih/minion/compare/master...brianmed:elected_recurring
Any thoughts on this code?

Recurring jobs could also be used for queue maintenance with Minion::repair. Which currently runs inside the main process and might therefore cause delays for job processing.

Ok, after thinking about this some more i believe that a complicated solution is not right for us. Instead we will focus on making Minion work better with cron. That means possibly adding a unique jobs feature if we can find a good enough solution.

Now that we have job dependencies, the old way of doing unique jobs has become much less interesting though. At the moment i can only really see expiring locks working there.
app->minion->add_task(some_task => sub {
  my ($job, @args) = @_;

  # Try to get the lock for up to 20 minutes (then it expires)
  return $job->finish('Another job already in progress for some_unique_key')
    unless $job->try_to_lock('some_unique_key', '20 minutes');

  ... do stuff ...

  # Release the lock in this job or a followup job (for job dependencies)
  $job->release_lock('some_unique_key');
});

I've been experimenting a bit with exclusive and shared locks. https://github.com/kraih/minion/compare/expiring_locks Sadly the shared lock part didn't work out too well, since on a busy queue a missing unlock might never resolve itself again, resulting in wasted resources. Maybe someone else has a better idea for how to do it.

Closing this issue, since from now on our priority will just be to make Minion work better with cron. Lets wait and see what features our users can come up with to make that happen.

@kraih I see a concurrency problem in current minion_lock function. Two or more queries can be executed simultaneously. We can get the unique_violation exception while insert or holders can be >= $3 while update.
What about using upsert for this case?
create function minion_lock(text, int, int) returns bool as $$
declare
  new_expires timestamp with time zone = now() + (interval '1 second' * $2);
  lock record;
begin
  delete from minion_locks where expires < now() or holders <= 0;
  insert into minion_locks as l values ($1, 1, new_expires)
    on conflict (name) do update
    set holders = l.holders + 1, expires = greatest(l.expires, new_expires)
    where l.holders < $3;
  return found;
end;
$$ language plpgsql;

Also you can simplify minion_unlock:
create function minion_unlock(text) returns bool as $$
declare
  new_holders int;
begin
  delete from minion_locks where name = $1 and holders <= 0;
  update minion_locks set holders = holders - 1 where name = $1 returning holders into new_holders;
  return found and new_holders >= 0;
end;
$$ language plpgsql;

@avkhozov I don't think it makes sense trying to salvage that code, it is conceptually broken, shared locks just can't be done that way.

Forgot to update this issue, we now have support for expiring locks. http://mojolicious.org/perldoc/Minion#lock


nvidia_x11.persistenced: fix hash

Motivation for this change
persistenced has a hash mismatch after download, and thus fails to build
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

@SuperSandro2000 If you're around could you take a look here? Thanks!

@SuperSandro2000 If you're around could you take a look here? Thanks!

Result of nixpkgs-review pr 108853 run on x86_64-linux 1

Result of nixpkgs-review pr 108853 run on x86_64-linux 1


Add Airdrop or Nearby share-like features to Hello

Is your feature request related to a problem? Please describe.
Add Airdrop or Nearby share-like features to Hello.
In the case of future needs, something like Airdrop will be the easiest way to transmit data without using a network hard drive. It can also be similar to Bluetooth without a mobile phone line, and the transmission will be more efficient than only Bluetooth transmission. fast; and he can also transfer files to multiple people, which is even more convenient.
Describe the solution you'd like
none
Describe alternatives you've considered
Snapdrop , More recommended, only it's a web page
Update: There is a desktop version, it just needs to be written.
OpenDrop , No interface and no updates in a year
Additional context
Nearby share

Airdrop

Snapdrop

Bluetooth file transfer (OBEX) would be an easier start and has a higher priority.


[Isogram] Implementation

All the Raku stuff looks good! I think all that's left is the Exercism stuff. The config.json file in the root directory needs updating. I think you may also need to run bin/configlet sync --update --docs --filepaths --metadata --yes --exercise isogram for the exercise documentation.


SBP: Control

What is the purpose of Nonce and OrgByNonce? They seem are not really necessary.
Why not include OrgConfiguration as attribute of Orgs ? (probably related to #53 )

check_membership is unnecessary. No need to create an extrinsic to read from the chain state. Extrinsics are meant to modify the chain state, not for querying.

Nonce and OrgByNonce where lookups to be able to access orgs without storing their hashes in the frontend. most of these inefficient storage issues root back in a time before we started to work with our graph implementation and are resolved now.
check_membership rooted in another topic and should not have been public, removed anyway as not required anymore.


Improve information sharing across SB diagnostics

Previous Stacked Borrows diagnostics were missing a lot of information about the state of the interpreter, and it was difficult to add additional state because it was threaded through all the intervening function signatures.
This change factors a lot of the arguments which used to be passed individually to many stacked borrows functions into a single DiagnosticCx, which is built in Stacks::for_each, and since it wraps a handle to AllocHistory, we can now handle more nuanced things like heterogeneous borrow of !Freeze types.

I was reminded that this can probably be used to produce better tag-tracking diagnostics. Might implement that later...

FYI this is now about a 10% perf regression on all benchmarks, on account of the requires_caller_location calls. Here's a profile of the backtraces benchmark if you're curious: flamegraph.svg.gz
I don't know why that function is so expensive, but the disassembly is massive.

Oh wow. Okay then let's do that yet, instead please open an issue to track further investivation.

Fresh brain in the morning. I realized that I was pointlessly checking for #[track_caller] on a lot of frames. I just amended with this diff:
diff --git a/src/helpers.rs b/src/helpers.rs
index e3142caa..bf536c24 100644
--- a/src/helpers.rs
+++ b/src/helpers.rs
@@ -923,6 +923,8 @@ impl<'a, 'mir: 'a, 'tcx: 'a + 'mir> CurrentSpan<'a, 'mir, 'tcx> {
             .unwrap_or(rustc_span::DUMMY_SP)
     }

+    // Find the position of the inner-most frame which is part of the crate being
+    // compiled/executed, part of the Cargo workspace, and is also not #[track_caller].
     fn current_span_index(tcx: TyCtxt<'_>, machine: &Evaluator<'_, '_>) -> usize {
         machine
             .threads
@@ -930,11 +932,11 @@ impl<'a, 'mir: 'a, 'tcx: 'a + 'mir> CurrentSpan<'a, 'mir, 'tcx> {
             .iter()
             .enumerate()
             .rev()
-            // Skip #[track_caller] frames
-            .filter(|(_, frame)| !frame.instance.def.requires_caller_location(tcx))
             .find_map(|(idx, frame)| {
                 let def_id = frame.instance.def_id();
-                if def_id.is_local() || machine.local_crates.contains(&def_id.krate) {
+                if (def_id.is_local() || machine.local_crates.contains(&def_id.krate))
+                    && !frame.instance.def.requires_caller_location(tcx)
+                {
                     Some(idx)
                 } else {
                     None

This is now on average perf-neutral, with some small improvements and regressions. requires_caller_location is now ~1% of cycles which is still oddly high but not concerning for us at the moment IMO.

(Please don't rebase+amend unless there are conflicts. Rebasing destroys being able to get decent diffs from github for what you changed between revisions.)

Please don't rebase+amend unless there are conflicts.

I was wondering how you were dealing with this before :upside_down_face:

For big PRs like this, ideally just keep adding new commits, then even after a rebase it's easy to see what is new. Squashing is an optimization that can happen when we are done. :)

:umbrella: The latest upstream changes (presumably #2488) made this pull request unmergeable. Please resolve the merge conflicts.

@rustbot author

I took the liberty to take care of the last nit, so that the nicer diagnostics can be part of the next Miri update. :)
@bors r+

:pushpin: Commit 0e5a53d12d5b845622afe495526b9622fc149930 has been approved by RalfJung
It is now in the queue for this repository.

:hourglass: Testing commit 0e5a53d12d5b845622afe495526b9622fc149930 with merge bf38971a08e0f072ca199f0840662fcf98174b16...

:broken_heart: Test failed - checks-actions

@bors r+

:pushpin: Commit 7397c8e9cf73ff338c0e00a64a4de66fd9553df3 has been approved by RalfJung
It is now in the queue for this repository.

:hourglass: Testing commit 7397c8e9cf73ff338c0e00a64a4de66fd9553df3 with merge 09118da54fa6498eef98533e2156d24958d4a79f...

:sunny: Test successful - checks-actions
Approved by: RalfJung
Pushing 09118da54fa6498eef98533e2156d24958d4a79f to master...

Thanks for taking care of this!


Persistent search

I tried adding the search addon to my editor. The normal search works fine but persistent search (Alt+F) does not. It works in the demo but not on my site.
Even if it did work, I would like the default (Ctrl+F) to be persistent search instead. Is it possible to configure this?
I'm using codemirror 5.38.0 with Vue. Chrome 67 on Ubuntu 18.04.

It works in the demo but not on my site.

You'll have to figure out the relevant difference, then.

I would like the default (Ctrl+F) to be persistent search instead. Is it possible to configure this?

Yes, bind that key combination to the "findPersistent" command, using extraKeys with a value like {"Ctrl-F": "findPersistent"}.

I found the extraKeys option hidden in the demo. Works perfectly now, thanks.


feat: override dynamodb config

Description
This PR add the possibility to override dynamoDB configuration via storage_options:

AWS_ENDPOINT_URL_DYNAMODB overrides AWS_ENDPOINT_URL,
AWS_REGION_DYNAMODB overrides AWS_REGION,
AWS_ACCESS_KEY_ID_DYNAMODB overrides AWS_ACCESS_KEY_ID,
AWS_SECRET_ACCESS_KEY_DYNAMODB overrides AWS_SECRET_ACCESS_KEY.

It solves cases where we use an S3-compatible storage without mutual exclusion but we want to use DynamoDB anyway as a locking provider.
It also solves the case where we want to pass different credentials or region for S3 and dynamoDB.
Related Issue(s)

closes #2497
closes #2287

Documentation
Documentation has been updated to explain how to override configuration.

@rtyler thanks for your quick review! I've just fixed a small equality in test. I've made a workaround before and I've left it in the code. It is now properly fixed. Thanks!

@thomas-chauvet @rtyler This might be short lived, if we can get delta-rs bumped soon to the new objectstore release. I would like us to default using putIfAbsent by default for any S3 implementation, then it's up to users who use S3 which does not support this, to set copyIfNotExists or allow unsafe_rename

Does this mean that locking provider will be definitely remove from delta-rs?

Does this mean that locking provider will be definitely remove from delta-rs?

Not per se, but the value of a locking provider is quite minimal unless you still use an S3 implementation like LakeFS that doesn't have these request headers yet

@ion-elgreco I don't expect the DynamoDB log store code to go away for at least 3-6 months, even if we had the putIfAbsent support today, I would want to keep the support around because existing deployments using DynamoDB will not all be ready to migrate 100%.
I see value in adding these parameters until such a time when all DynamoDB code is removed from delta-rs, which currently has no scheduled time.

@ion-elgreco I don't expect the DynamoDB log store code to go away for at least 3-6 months, even if we had the putIfAbsent support today, I would want to keep the support around because existing deployments using DynamoDB will not all be ready to migrate 100%.
I see value in adding these parameters until such a time when all DynamoDB code is removed from delta-rs, which currently has no scheduled time.

For sure, we should keep it for some time!

Thanks for the merge!


Doesn't work :(

Looks like this might not be working anymore. Tried in DMs and on a public server!

Yeahh tiktok seems to really not like bots. I'll have to take another look at this over the weekend when I'm free :disappointed:

Should be working again. Looks like tiktok is load balancing between a backend that runs on nextjs and https://github.com/sigi-framework/sigi so requests that go to the wrong backend randomly fail if not dealt with properly lol

Awesome! Does this by chance work with the share links that have the subdomain vm. in front of them?
For example https://vm.tiktok.com/ZMLRMM4KU/?

Nevermind it does when you put https://vm.fixtiktok.com! This is awesome!

Nevermind it does when you put https://vm.fixtiktok.com! This is awesome!

this appears top be broken now
https://vm.tiktok.com/ZMYD2yJvK/
https://vm.fixtiktok.com/ZMYD2yJvK/

Part of this problem is Tiktok's video links don't end with an mp4 and that seems to be a criteria for Discord to embed videos in opengraph tags. I solved that by doing an internal redirect from an mp4 fixtiktok link to tiktok's regular video link and that worked for a while. But it seems discord changed how their redirects coming from opengraph endpoints are followed and it stopped working some time ago and I haven't been able to get it working since.
I can try taking a second look but I doubt they've changed that behavior. This could be fixed by directly streaming the video from the fixtiktok link but I'm not really interested in that since it would be very costly.

the domain just seems to be down

Maybe they changed it. When I first made it the only way to get it to work was with a redirect. I'll take another look


Add missing sponsored rel

This PR adds a missing sponsored rel for the "special sponsor".
Related PR with previous changes: https://github.com/vuejs/vuejs.org/pull/2316

Thanks!


THC TDA

Adds thc.py introducing the new function of completing TDA calculations using THC integrals. Using this method should scale as N^3 in comparison to the previous Cholesky decomposition method (scales as N^4)
Current implementation requires the THC tensors to be calculated from a different source and then imported into the code. This is currently achieved within the ints.py. Future implementations will hopefully directly interface with a method to calculate THC ERIs.

Plan after call:

thc_opts only additional keyword argument in BaseGW, for now only contains filepath
For now, just have momentGW/thc.py that contains a new Integrals and TDA class that inherit from the standard versions
Specify THC via polarizability i.e. polarizability="thc-dtda", which controls what is returned by GW.ao2mo and GW.build_se_moments
Move the fock.py Fock matrix building functions inside the Integrals classes - this allows us to also remove the calls to PySCF in build_se_static in place of these functions, so we can maintain scaling with THC

I think this should make everything interface nicely without chaning much existing code

I've got a few tweaks to do to this, but apart from that the main things that are missing now are:

[ ] Tests - we need a smaller test system
[ ] THC Fock build (also needed for build_se_static)

after this we should be good to merge

I've changed some stuff around, made variable names more consistent with the DF implementation, and I think things are looking ok now. Still need tests, and to stress test the miscellaneous features in the GW solver. Then self-consistency and parallelism.
Not sure what our merge policy here is - perhaps this should just live in a development branch until we have a usable interface with the THC integral generation.

Can we move the MPI stuff to a new branch? Then we can look to getting this PR merged this week, I think it's ready to go

Branched MPI changes and reverted to pre MPI

on second thought, maybe we want to get the self-consistency working before merging

LGTM, great stuff @mkakcl


optional server response analyse functions

this should make it so easy to customize getting tokens and exp, that all types of backends should be supported

closes https://github.com/SpoonX/aurelia-authentication/issues/231

Must be rebased

Rename last two commit messages. One has a quote, the other is missing a space.

Feedback, but after 👍


[FEATURE]  I will work on Header since it's alignment is not proper i will try to make it better.

Description
No response
Screenshots
No response
Any additional information?
No response
What browser are you seeing the problem on?
No response
Checklist

[X] I have checked the existing issues
[X] I have read the Contributing Guidelines
[X] I am willing to work on this issue (optional)
[X] I am a GSSoC'23 contributor

Go ahead @Aakaassh, you can work on it.
Also, just to clarify you've marked yourself as a GSSoC'23 contributor.
But since the GSSoC'23 contribution period is over now, your contribution will not considered for that.
Otherwise, your contributions are always welcome.

I would like to work on this, please assign it to me

Closing this because the issue is already resolved.


Vld records marked as deleted  are not downloaded by CSVProc

@geoffj-FUG reports that If a record has an 'x' as a flag, the record does not get downloaded into the csv file. FreeCEN1 allowed the flags to be uploaded but they are no longer used by reports. CSVProc requires them to be resolved or replaced by an entry in the Notes field. This appears to be the cause of the missing records.

@geoffj-FUG I can confirm that the download does not download an entry that contains the deleted flag. It is NOT clear to me what action you want us to take.
From my research both FC1 and FC2 simply ignore the flag and treat it like any other record. In your file the first 2 deleted flag entries are in the same household. See attached.

A search on FC1 produces the following result

A search on FC2 produces the same result.

In both cases the record with the deleted flag is treated like any other record; i.e the flag is ignored.
So what should we be doing on the download? It is not clear to me.
It seems the record should be included. But should we add something to the notes ?
As far as I am aware there is no deleted field in CSVProc.

@geoffj-FUG since there has been no advice yet and in order to make progress on the other aspects of the VLD download I propose to include the entries in the download and set the uninhibited flag to n and added a note "Deleted flag set on VLD; "

@geoffj-FUG Using n is not going to work. Will try x

Kirk
The deleted flag is not that flag that is causing the problem. It is the flags that were put into the following columns in the FC1 tarditional spreadsheet - H (address flag and uninhabited etc. Uninhabited etc download OK), K (Names flag), P (Age etc flag), S (Occupation flag) and V (POB flag).
It is the x flags  in FC1 tht drew the attention of the proofreader / validator to a problem record.
In FC2 we require this flag to be replaced by a Note and for it to be removed before a piece can be Incorporated. In FC1 the flags were left if the issue was not resolved. It is these unresolved x flags that are causing the problem.
Geoff

@geoffj-FUG sorry to disagree but the first problem is the deleted flag from the VLD entry.
The code as originally written did not output any entry with the deleted flag field in the vld entry set to D.
A vld entry has 299 bytes and is set up as
Entry field	Transcriber field	Starting_byte	Number_of_bytes
A	X	0	1	Deletion marker (D or blank)
B	X	1	4	Not used. Was Registration district - this usage is discontinued.
C	X	5	6	A six digit number (leading zeros) which counts the households
D	X	11	4	A four digit number (leading zeros) which counts members in each household
E	A	15	20	Parish name (I do not check this, it is up to you to get it right!)
F	B	35	4	Enumeration district (3n+1a, the remaining numeric fields have trailing spaces)
G	C	39	5	Folio number (4n+1a)
H	D	44	4	Page number (4n)
I	E	48	4	Schedule number (3n+1a)
J	F	52	5	House number (4n+1a)
K	G	57	30	House/Street name (default -)
L	H	87	1	Uninhabited flag (b, u, v, n, x or -)
M	I	88	24	Surname (capitals, default -)
N	J	112	24	Forenames (default -)
O	K	136	1	Flag for name fields (x or -)
P	L	137	6	Relationship (default -)
Q	M	143	1	Condition (M, S, U, W or -)
R	N	144	1	Sex (M, F or -)
S	O	145	3	Age (no default but 999=unknown/unreadable)
148	1	Age unit(y, m, w, d or -)
T	P	149	1	Flag for detail fields i.e. rel/cond/sex/age (x or -)
U	Q	150	30	Occupation
R				Employed Status (extracted from occupation field)
V	S	180	1	Flag for occupation (x or -)
W	T	181	3	County code (3a capitals, no default but UNK if not known)
X	U	184	20	Birth place (default -)
Y	V	204	1	Flag for birth place (x or -)
Z	W	205	6	Disability (default blank)
AA	X	211	1	Language (W, E, B, G or blank)
AB	Y	212	44	Notes (default blank, no case)
AC	X	276	3	Alternate birth county
AD	X	279	20	Alternate birth place
As I said earlier From my research both FC1 and FC2 simply ignore the deleted flag and treat it like any other record.
Some of those records have other flags set but not very many. I have extracted the 552 vld entries in file RG10_2349 and listed them in the attached file so you can see what I am talking about.
vld deleted entries.xlsx
So do we just ignore the D flag and only may attention to the other flags (There is already code dealing with some of these). Do we add a message into the notes. What do we do?

@FreecenBren has confirmed as follows:
So I used the search for this PIECE I found where a D has been used.
You will see that at some stage the D was meant to be deleted but it was not.
The system though still passed it.

Excellent work Brenda.
You have confirmed that FC1 paid no attention to the deleted flag in the VLD and included that record in the database.
I have confirmed that FC2 also ignored the deleted flag. Hence the 2 databases are consistent but incorrect.
You have confirmed in that one case that the record with the deleted flag should not have been included.
Perhaps Geoff can confirm the situation with his example.
It means that the download of a vld file in CSVProc format should not download a record with the VLD deleted flag unless we want to flag it in some other way.
That leaves us with the question of what to do about it. FC1 cannot be corrected.  FC2 can be corrected by writing a database correction task, having first confirmed the extent of the problem.

We could, well I will anyway, check all  of the VLD’s  I still have to
finish off, before they are added to the database, to ensure that all D’s
are amended.
Not sure about checking those already online, but if you can identify any
then I will do them,  no problem, as long as they are for my counties.
(Still plenty of those)
Cheers Brenda
On Mon, 29 Nov 2021 at 18:58, Kirk Dawson @.***> wrote:

Excellent work Brenda.
You have confirmed that FC1 paid no attention to the deleted flag in the
VLD and included that record in the database.
I have confirmed that FC2 also ignored the deleted flag. Hence the 2
databases are consistent but incorrect.
You have confirmed in that one case that the record with the deleted flag
should not have been included.
Perhaps Geoff can confirm the situation with his example.
It means that the download of a vld file in CSVProc format should not
download a record with the VLD deleted flag unless we want to flag it in
some other way.
That leaves us with the question of what to do about it. FC1 cannot be
corrected. FC2 can be corrected by writing a database correction task,
having first confirmed the extent of the problem.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/FreeUKGen/FreeCENMigration/issues/1333#issuecomment-981922456,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADIL3VQUFPAKYJYKW3DAACLUOPEN5ANCNFSM5ING4P3Q
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

@PatReynolds  to research the utility to researchers of retaining the Deleted entries, and report back.

Just checking: D means there was an entry in the census, which was struck through?  There are some examples mentioned in this online discussion: https://www.rootschat.com/forum/index.php?topic=791507.9
Thanks,
Pat

Pat
The old FC1 Wincc and Valdrev had a field that kept track of the progress on the piece for the volunteer. This had several codes including ‘D’.
The ‘D’ was generated by the software when an entry was deleted by the Proofreader (WinCC) or the Validator (Valdrev). It was a soft delete.
There was no instruction apparent to me that the Coordinator needed to remove these records post validation so my presumption has always been that these records were deleted on upload but obviously they were not.
Neither did FC1 seem to deal with them in any way. It seems to have been an idea that was sound but not finalised. Hence the current problem of deleted records appearing in the FC2 database.
The intention seems to be that these records were not needed for a variety of reasons and therefore would not be displayed.
From the actions that I have taken in the past I see no issue in simply removing them from the database. They are and always have been intended as deleted records.
Geoff
From: PatReynolds @.>
Sent: Thursday, 27 January 2022 3:55 AM
To: FreeUKGen/FreeCENMigration @.>
Cc: geoffj-FUG @.>; Assign @.>
Subject: Re: [FreeUKGen/FreeCENMigration] Vld records marked as deleted are not downloaded by CSVProc (Issue #1333)
Just checking: D means there was an entry in the census, which was struck through? There are some examples mentioned in this online discussion: https://www.rootschat.com/forum/index.php?topic=791507.9
Thanks,
Pat
—
Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/1333#issuecomment-1022446149 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AKCPIFMUAKVKWBMMKKXJHCTUYAYPRANCNFSM5ING4P3Q .
Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675  or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub .
You are receiving this because you were assigned.  https://github.com/notifications/beacon/AKCPIFMLEUACFNI2OUCCIRTUYAYPRA5CNFSM5ING4P32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOHTYUURI.gif Message ID: @.*** @.***> >

At this point I am raising a BIG RED FLAG.
I have just downloaded a vld file. It is RG9 1705 Bedminster. I noticed that there is a Note in the Notes column that the vld flag is set to D. Soo I started looking at the entries.
Take Jane Brice F3 P1 Sch 2. She is on the image and should not be deleted. Yet she had a D. (I am afraid that the vld file is now in the attic as I was in the process of replacing it with a csv file).
So I have had a look at the next few entries. They are all valid entries, not deleted ones.
There were 470 entries with this comment. The number of soft deletions in a piece was typically less than 5.
This disproves the hypothesis that they are flags carried through  from the old WinCC,  Valdrev  system.
Geoff

So the next task is to understand why the validator or coordinator marked entries as D (if not because the enumerator had deleted them).

Pat
The Proofreader or Validator would not have added the D. The system would have.
The piece that I looked at was one that I would have validated many years ago. I can see absolutely no reason for the flags to be in the file.
What I have found disproves the hypothesis that the flags were entered by Wincc or Valdrev.
The answer lies somewhere in FC1 itself.
Geoff

Concluded that there is no reason why the D is there, so @Captainkirkdawson will remove them.

Records with a soft delete are now downloaded and a note added. Deployed to test3

All of the reported issues are now correct in a download. I am happy for this to go live.
Geoff


MM 14040 : update marked for parentheses in link url

Summary
Modifies the href regex pattern search to include embedded parentheses
e.g. [hmmm](https://somewhere/?param=(currentUser()))
Jira
Fixes https://github.com/mattermost/mattermost-server/issues/10407
other repositories
Supporting https://github.com/mattermost/mattermost-webapp/pull/2814

I am experiencing one error in npm run test for mattermost-webapp with this change.
    Expected value to be equal to:
      "
test.:test
"
    Received:
      "
test.:test
"

I will look into this more.

The above error occurs on master and is not related to the change in this PR

Please let me know if I could make this fix as well in this PR so as to be able to pull it into webapp.  :)

@hmhealey Can you help with this?

@scottleedavis If you update your webapp branch off the latest master, mattermost/mattermost-webapp#2842 was just merged which modifies that test to support some recent changes made to our marked fork

@hmhealey I've updated the regex in a way I am not sure is idiomatic, however all the tests pass and the original issue url of https://somewhere/?param=(currentUser()) parses correctly, as well in the webapp markdown example.

Unfortunately, this still has issues with certain bad URLs. This pattern has issues because the [^*] in \([^*]*\) can match opening and closing brackets which leads to bad behaviour with a string like [link]((((((((((((((((((((((((((((((((((((((((((((((((((((().
Without ditching regexes entirely, I think our only option would be to support brackets nested one level deep so ?params=(currentUser()) would work, but ?params=(currentUser(id())) would not. It's impossible to create a regex that supports arbitrarily nested brackets, so that's the best we can do without creating more and more complex regexes

Ok updated with my best effort to only do one level deep as requested /(?:[^()]|\([^()]*\)|\S*\(?\([^()]*\)\))*/.  Happy to improve upon it or try something else.  Tested that your second example ?params=(currentUser(id())) does not match.

@hmhealey Would you please review again?

I added 14 tests for the nested urls.  Do you recommend I change anything or make additions?

Looks good. I tried to see if there were any more cases that caused catastrophic backtracking with this, but it seems solid to me.

Now that this is merged, would you mind submitting a PR to the web app branch to update its version of the library?


style TopBar

as per Figma

@amaurymartiny I've branched from your now merged branch thus a bit messy


Use new grid layout

There a significant visual difference, so i'm not sure this is ready to merge, but I think theres enough to be worth discussion. I'd recommend looking at the commits (and their messages) in isolation, rather than the whole diff.
Mostly this converted fine, there were some sticking points around:

Needing an element to wrap the filters and results columns, acting as a grid row - basically not the header. I chose .filtering as a not-awful start, but I think we could do better.
Continuing to enforce the max-width of the header, to ensure the line length doesn't go beyond 2/3rds, is done in a fiddly way. It requires floating
One option would be to it in a row/column, but that would add quite a bit of markup just for CSS hooks.
Another would be using max-width
Finally, the show stopper; the gutters between columns is significantly smaller, 30px in the new vs 60px in the old. As the screenshots show, it's a hefty change.

The first point is semantic flddling, while the other two are a class of problem we'll see on other applications when trying to use modern grids and so are worth proper discussion.
Screenshots!

Before
After

Good PR :+1:. Not sure if the gutter spacing is a show stopper or we should hack the spacing back in. Thoughts @futurefabric?

There's less visual separation with that tighter gutter, but I don't think this should prevent merging.
We can always take a look at it separately later if needs be.

Consensus seems to be that the smaller gutter is fine. It's the standard way the new grid works, which is used on frontend.
I've just spotted a bug with the beta label layout, which i'll fix, and then this should be good to merge.

Updated to handle cases with; related links, beta and email links:

One totally minor feedback point. Other than that this is a good change :+1: Lots of red.

Fixed the clear, good catch!


some mistakes when I downloading it

When I run "go get -d github.com/itlabers/ofd-go"
I got this error: "module declares its path as: ofd but was required as: github.com/itlabers/ofd-go"
Thanks for your job about OFD . It seems the first Go version for OFD standard .

Thank you for your feedback.
Golang  module  name  has been modified,you can try again.

it works now . thanks.


Confusion about parsing of "not" operator

Hi! In the following example I'd expect "not" to be a first-class member of the AST. But it appears to just be concatenated with the operator after it. This would imply I've got to then do string parsing of the operation field if I'm trying to discover "not" use:
> const { statement: [{ where: [where5] }] }  = parser('select 1 where processName =  \'/bin/sh\' and id not like \'foo\'');
undefined
> where5
{ type: 'expression',
  format: 'binary',
  variant: 'operation',
  operation: 'and',
  left:
   { type: 'expression',
     format: 'binary',
     variant: 'operation',
     operation: '=',
     left: { type: 'identifier', variant: 'column', name: 'processname' },
     right: { type: 'literal', variant: 'text', value: '/bin/sh' } },
  right:
   { type: 'expression',
     format: 'binary',
     variant: 'operation',
     operation: 'not like',
     right: { type: 'literal', variant: 'text', value: 'foo' },
     left: { type: 'identifier', variant: 'column', name: 'id' } } }

Is this intentional? I'd expect the AST to look more like:
  ...
  right:
   { type: 'expression',
     format: 'unary',
     variant: 'operation',
     operation: 'not',
     expression:
     {  type: 'expression',
        format: 'binary',
        variant: 'operation',
        operation: 'like',
        right: { type: 'literal', variant: 'text', value: 'foo' },
        left: { type: 'identifier', variant: 'column', name: 'id' } } } }

Here I have an example:
select (not 1) + 1, not (1 + 1), not 1 + 1

If you write it in sqllite http://sqlfiddle.com/#!7/9eecb7/7512 you can see that not 1 + 1 means not (1 + 1) but not (not 1) + 1

Here I have an example:
select (not 1) + 1, not (1 + 1), not 1 + 1

If you write it in sqllite http://sqlfiddle.com/#!7/9eecb7/7512 you can see that not 1 + 1 means not (1 + 1) but not (not 1) + 1


Reddison Fuzzy matching iteration is too time-consuming

Expected behavior
Actual behavior
When I use the keys method for fuzzy matching, there are only 15 keys, which takes 7-800 milliseconds after iterative processing
Steps to reproduce or test case
RedissonClient.keys("server:*").iterator()
Redis version
5.0
Redisson version
redisson-all-3.15.0
Redisson configuration

Unable to reproduce.
        for (int i = 0; i < 15; i++) {
            redisson.getBucket("test" + i).set(i);
        }

        long t = System.currentTimeMillis();
        Iterable<String> keys = redisson.getKeys().getKeysByPattern("test*");
        assertThat(keys).hasSize(15);
        System.out.println(System.currentTimeMillis() - t);

It took 47ms.


new image for Matt Zaiger

More like me

You also need to change the filename in your profile file in team/_posts/ to point to this new image

OK, I merged this and linked the image


Container opencost keeps exiting with error code 2 when trying to hit RateCard API

Setup

We have an AKS private cluster running on which opencost is deployed using the helm-chart here
The secrets service-key.json, cloud-integration.json along with the service principal and associated custom roles were created as specified here

Issue

Upon trying to access the "Cloud Costs", seeing a "502 Bad Gateway" error in the UI
The opencost container keeps restarting with exit code 2 and seems to not proceed further after the log
2023-12-21T09:52:38.134643073Z INF Using ratecard query OfferDurableId eq 'MS-AZR-0003p' and Currency eq 'USD' and Locale eq 'en-US' and RegionInfo eq 'US'

Is this still an issue? Were you using 1.108?


RFC extend Box in BorderBox, remove default bg

Was chatting with @broccolini today and she mentioned that BorderBox should extend Box & that we shouldn't be setting a default background color, which makes sense to me, so I made those changes here!
🚨 breaking change! 🚨

bg no longer has a default value

Merge checklist

[ ] Changed base branch to release branch
[ ] Tested in Chrome
[ ] Tested in Firefox
[ ] Tested in Safari
[ ] Tested in Edge

@shawnbot I'm not sure if switching over to styled-components fixes this problem, but we can try! Are you thinking of using the styled-components version of system-components or just nixing system-components all together and using styled-components + styled-system directly within component files?

also update - weirdly the styles are not getting duplicated all of the time... which makes me think it might be related to SSR in some way. On first load there's no duplication, but once you refresh the page the styles get duplicated.

@shawnbot yeah I'm thinking we need to switch to @rebass/components and see what that fixes, we have a few band-aids in withSystemProps that I'm hoping we can get rid of with @rebass/components. I'd like to get an idea of how much refactoring would be required first though before starting... so far I think the main difference is is is now extends.
My only other worry is I'm not sure if that package is going to be maintained in the future or not, since they moved Rebass off of it :/ see https://github.com/rebassjs/components/issues/4
Since styled-components has as I wonder if switching to styled-components would mostly alleviate our need for system-components/rebass components?

@shawnbot I'm going to merge this as is - and follow up on the duplicated styles issues + start looking into moving away from rebass/components / system-components


enable keycload / ocp sso instead of dex

Enable OAUTH in ArgoCD 1.2
oc -n openshift-gitops patch argocd openshift-gitops --type='json' -p='[{"op": "add", "path": "/spec/sso", "value": {"provider": "keycloak"} }]'

Enable the admin role in ArgoCD
oc patch cm/argocd-rbac-cm -n openshift-gitops --type=merge -p '{"data":{"policy.default":"role:admin"}}'

Included in https://github.com/rcarrata/devsecops-demo/commit/5dd935184cbd3ba80a68d7551ff2a88ffb0f693a

Working as expected

Everything working!


Add qglviewer Qt4 rosdeps for Ubuntu yakkety and zesty

can you add a rule for debian stretch as well ? Thanks!

I think this is failing because platforms have to be ordered alphabetically


4.4.1: Timeout for lots of torznab search results.

Smaller pockets continue to work, seems if there's a lot of results it times out for all.
info: [torznab] Searching 1 indexers for Oatmeal.For.Me.S01.2160p.DSNP.WEB-DL.DDP5.1.Atmos.DV.HDR.H.265-CRFW
error: snatching http://jackett.jackett:9117/dl/broadcasthenet/?jackett_apikey=[REDACTED]&path=different&file=Oatmeal.For.Me.S01.HDR.2160p.DSNP.WEB-DL.DDPA5.1.H.265-SKGTV timed out
error: snatching http://jackett.jackett:9117/dl/beyond-hd-api/?jackett_apikey=[REDACTED]&path=different&file=Oatmeal.For.Me+S01+2160p+DSNP+WEB-DL+DDP+5.1+Atmos+DV+HDR+H+265-CRFW+(Dolby+Vision+%2F+HDR10) timed out
error: snatching http://jackett.jackett:9117/dl/broadcasthenet/?jackett_apikey=[REDACTED]&path=different&file=Oatmeal.For.Me.S01.DV.2160p.DSNP.WEB-DL.DDPA5.1.H.265-CRFW timed out
error: snatching http://jackett.jackett:9117/dl/torrentleech/?jackett_apikey=[REDACTED]&path=different&file=Oatmeal.For.Me+S01+2160p+DSNP+WEB-DL+DD%2B+5.1+Atmos+DoVi+HDR+HEVC-SiC timed out
error: snatching http://jackett.jackett:9117/dl/broadcasthenet/?jackett_apikey=[REDACTED]&path=different&file=Oatmeal.For.Me.S01.DV.2160p.DSNP.WEB-DL.DDPA5.1.H.265-MZABI timed out
error: snatching http://jackett.jackett:9117/dl/blutopia-api/?jackett_apikey=[REDACTED]&path=different&file=Oatmeal.For.Me+S01+2160p+HTSR+WEB-DL+DD%2B+5.1+Atmos+PQ10+H.265-SKGTV timed out
error: snatching http://jackett.jackett:9117/dl/beyond-hd-api/?jackett_apikey=[REDACTED]&path=different&file=Oatmeal.For.Me+S01+2160p+DSNP+WEB-DL+DDP+Atmos+5.1+DV+HEVC-MZABI timed out
error: snatching http://jackett.jackett:9117/dl/filelist/?jackett_apikey=[REDACTED]&path=different&file=Oatmeal.For.Me.S01.2160p.DSNP.WEB-DL.DDP5.1.Atmos.DoVi.H.265-SiC timed out
error: snatching http://jackett.jackett:9117/dl/blutopia-api/?jackett_apikey=[REDACTED]&path=different&file=Oatmeal.For.Me+S01+Hybrid+2160p+DSNP+WEB-DL+DD%2B+5.1+Atmos+DV+HDR+H.265-SiC timed out
error: snatching http://jackett.jackett:9117/dl/torrentleech/?jackett_apikey=[REDACTED]&path=different&file=Oatmeal.For.Me.S01.2160p.DSNP.WEB-DL.DDP5.1.Atmos.DV.HDR.H.265-CRFW timed out
error: snatching http://jackett.jackett:9117/dl/aither-api/?jackett_apikey=[REDACTED]&path=different&file=Oatmeal.For.Me+S01+2160p+DSNP+WEB-DL+DD%2B+5.1+Atmos+DV+HDR+H.265-SiC timed out
error: snatching http://jackett.jackett:9117/dl/bithdtv/?jackett_apikey=[REDACTED]&path=different&file=Oatmeal.For.Me+S01+INTERNAL+HDR+2160p+WEB+H265-SKGTV timed out
error: snatching http://jackett.jackett:9117/dl/uhdbits/?jackett_apikey=[REDACTED]&path=different&file=Oatmeal.For.Me+S01+%5B2019%5D+2160p+WEB-DL+Scene+-SKGTV timed out
error: snatching http://jackett.jackett:9117/dl/morethantv-api/?jackett_apikey=[REDACTED]&path=different&file=Oatmeal.For.Me.S01.DV.2160p.DSNP.WEB-DL.DDPA5.1.H.265-SiC timed out
error: snatching http://jackett.jackett:9117/dl/morethantv-api/?jackett_apikey=[REDACTED]&path=different&file=Oatmeal.For.Me.S01.2160p.DSNP.WEB-DL.DDP5.1.Atmos.DV.HDR.H.265-CRFW timed out
error: snatching http://jackett.jackett:9117/dl/hawke-uno/?jackett_apikey=[REDACTED]&path=different&file=Oatmeal.For.Me+(2019)+S01+(2160p+DSNP+WEB-DL+Hybrid+H265+DV+HDR+DDP+Atmos+5.1+English+-+HONE) timed out
error: snatching http://jackett.jackett:9117/dl/morethantv-api/?jackett_apikey=[REDACTED]&path=different&file=Oatmeal.For.Me.S01.INTERNAL.HDR.2160p.WEB.H265-SKGTV timed out
error: snatching http://jackett.jackett:9117/dl/aither-api/?jackett_apikey=[REDACTED]&path=different&file=Oatmeal.For.Me+S01+2160p+DSNP+HDR+WEB-DL+DD%2B+5.1+H265-SKGTV timed out
error: snatching http://jackett.jackett:9117/dl/blutopia-api/?jackett_apikey=[REDACTED]&path=different&file=Oatmeal.For.Me+S01+2160p+DSNP+WEB-DL+DD%2B+5.1+Atmos+DV+H.265-MZABI timed out
error: snatching http://jackett.jackett:9117/dl/blutopia-api/?jackett_apikey=[REDACTED]&path=different&file=Oatmeal.For.Me+S01+2160p+DSNP+WEB-DL+DD%2B+5.1+Atmos+HDR+H.265-SKiZOiD timed out
error: snatching http://jackett.jackett:9117/dl/hawke-uno/?jackett_apikey=[REDACTED]&path=different&file=Oatmeal.For.Me+(2019)+S01+(2160p+DSNP+WEB-DL+H265+DV+HDR+DDP+Atmos+5.1+English+-+CRFW) timed out
error: snatching http://jackett.jackett:9117/dl/thesceneplace/?jackett_apikey=[REDACTED]&path=different&file=Oatmeal.For.Me+S01+2160p+DSNP+WEB-DL+DD%2B+5.1+Atmos+DV+HDR+H.265-CRFW timed out
info: [server] Found 0 torrents for { infoHash: '30e3c19bf3bdede2585a55ef4424d5eeb227ef21' }

I had the same issue, raised the timeouts from 30 to 60 seconds solved that for me.

ATM It's not a config option, I changed it in the source.

I am also having this issue.

Can confirm I've had this issue since 4.4 release

I've also been having this issue; BaukeZwart's solution helped in most cases, though there are still lots of timeouts when there's any kind of load on the disk (e.g. while downloading torrents). Would be nice to be able to play with different timeouts in the config but increasing to 60 seconds does help.

I also saw a few situations where 60 sec wasn't enough. Although they were rare, I'm going to try 75 sec next.

Jeez. yeah, still seeing this even with the delay from bz.

I increased the delay to 90 seconds.

just chiming in to +1
effecting me more frequently than is ideal as well.

I increased the delay to 90 seconds. It still happens but very rarely.

Does it make sense to be in the config? Im having same issue here.

I increased the delay to 90 seconds. It still happens but very rarely.

Does it make sense to be in the config? Im having same issue here.

I've had to increase mine to 120. Ideally it wouldn't be a shared timeout for them all, but for now increasing this is the "cheap fix". At the end of the day, increasing the value is really just papering over the problem, including the global "delay".  These should be configured per torznab endpoint, whether it's the /name that jackett gives, or the random /id that prowlarr gives.

These should be configured per torznab endpoint, whether it's the /name that jackett gives, or the random /id that prowlarr gives.

agreed, although remember that just increasing the global timeout to an arbitrarily large number is equivalent to cross-seed's behavior pre 4.4. Per-indexer timeouts is probably something we'll only be able to support once cross-seed has a web ui.


Add command line flag to enable Key Manager API

First PR that starts introducing the Key Manager API as discussed in #439
This introduces the boolean command line flag that controls whether or not to enable the new API.

Your build is failing due to the formatting errors in the spotless check. You can run ./gradlew spotlessApply to fix the formatting issue.

Should have ran it after the string change 😅 Done.


[1.3][Energy EVSE] NumberOfWeeklyTargets and NumberOfDailyTargets attributes has to be removed from the All-clusters-app as per spec changes

Reproduction steps
Below mentioned attributes has been removed from the spec in this commit (43d5437f427fa0befff1c58980e9137055e9c1b9) . So need to remove those attributes from the Chip-tool and All-clusters-app.

NumberOfWeeklyTargets
NumberOfDailyTargets

Specification Reference: https://github.com/CHIP-Specifications/connectedhomeip-spec/blob/master/src/app_clusters/EnergyEVSE.adoc#attributes
GitHub hash of the SDK that was being used
7381d7635789a87d9a37b105838be5ee797c21d0
Platform
raspi
Platform Version(s)
RPI - 4, 8GB RAM
Type
Spec Compliance Issue

Issue resolved in this PR: https://github.com/project-chip/connectedhomeip/pull/31406


[#44] Added support for missing dependencies for sibling project

Thin jar includes sibling projects jars. It also adds dependencies of the sibling projects (Fixes #44 ).
Added support for classifier in the dependencies (Fixes #43 ).

Can these changes go into main branch?

Hi @balbusm yeah trying to find time to assess. Will do this as soon as I can.


Package Manager Product Definitions evolution proposal

Third version of evolution proposal, updated based on feedback.

Thanks for your comments, @ddunbar.  I will update the draft based on your feedback, and then we can hopefully start the review later today.


Q: c2goasm

Sorry for posting here but c2goasm does not have enabled issues and this seems like the "main" project here.
I just found the c2goasm project by accident and am wondering if it is capable for making ffmpeg library go-native?

No, that is not going to work, that is much too big a library for it.
Generally speaking c2goasm can be used to port "leaf" functions to golang assembly, meaning functions that do not  call other functions (or fully inline them). Also functions that would malloc()/realloc() structures etc. will not work as the C memory management is not compatible with Golang's garbage collector.
So the best you may be able to do is port small bits and pieces to Go, but that would still leave a lot of code to be ported in another fashion.


Show deploy progress in git push

We should print the deploy events that happen during the last part of a git push like we do in flynn scale.

@titanous you mean that instead of:
=====> Waiting for web job to start...
=====> Default web formation scaled to 1

you want something like:
15:37:48.001 ==> web host-692d8a7f-3ae4-4ad9-80ba-634e521225e2 starting
15:37:48.906 ==> web host-692d8a7f-3ae4-4ad9-80ba-634e521225e2 up

Yeah.

Flynn is unmaintained and our infrastructure will shut down on June 1, 2021. See the README for details.


Pre-trained Reward Models

Hello! I was wondering if it would be possible to release the pre-trained reward models in addition to the codec and video models that are already published (https://huggingface.co/tauhuang/diffusion_reward/tree/main).

We use codec and video models as reward models. So downloading these models is equivalent to getting the corresponding reward model.

Closing because of inactivity.


Improves documentation

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (d93af40) 81.76% compared to head (e1d9541) 81.77%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #66      +/-   ##
==========================================
+ Coverage   81.76%   81.77%   +0.01%
==========================================
  Files          56       56
  Lines        7452     7452
==========================================
+ Hits         6093     6094       +1
+ Misses       1359     1358       -1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


exthost.bufferUpdate

                                                                                indentation.detectAndSet
                                                                                feature.syntax.bufferUpdate

                                                                                [INFO]     +0ms Oni2.Store.dispatch : (BufferSetIndentation (4, <opaque>))
                                                                                [DEBUG]    +0ms Oni2.Store.dispatch : After: (BufferSetIndentation (4, <opaque>))
                                                                                [DEBUG]    +0ms Oni2.Store.StoreThread : Effect complete: Batch:
                                                                                Batch:
                                                                                exthost.bufferUpdate

                                                                                indentation.detectAndSet
                                                                                feature.syntax.bufferUpdate

                                                                                [DEBUG]    +0ms Oni2.Store.StoreThread : Running effect: Batch:
                                                                                languageFeature.checkForDefinition

                                                                                [DEBUG]    +0ms Oni2.Store.LanguageFeatures : Checking for definition...
                                                                                [INFO]     +0ms Oni2.Model.LanguageFeature:Oni2.DefinitionProvider :
                                                                                Querying provider: exthost.10
                                                                                [DEBUG]    +0ms Oni2.Model.LanguageFeature:Oni2.DefinitionProvider :
                                                                                Provider skipped: exthost.9
                                                                                [INFO]     +0ms Oni2.Model.LanguageFeature:Oni2.DocumentHighlightProvider :
                                                                                Querying provider: exthost.14
                                                                                [DEBUG]    +0ms Oni2.Model.LanguageFeature:Oni2.DocumentHighlightProvider :
                                                                                Provider skipped: exthost.13
                                                                                [DEBUG]    +0ms Oni2.Store.StoreThread : Effect complete: Batch:
                                                                                languageFeature.checkForDefinition

                                                                                [DEBUG]    +0ms Oni2.Store.StoreThread : Running effect: Batch:
                                                                                scm.getOriginalUri
                                                                                indentation.detectAndSet
                                                                                title.update

                                                                                [INFO]     +0ms Oni2.Store.dispatch : (BufferSetIndentation (4, <opaque>))
                                                                                [DEBUG]    +0ms Oni2.Store.dispatch : After: (BufferSetIndentation (4, <opaque>))
                                                                                [INFO]     +0ms Oni2.Store.dispatch : (SetTitle "KanbanBoard.js - client - Onivim 2")
                                                                                [DEBUG]    +0ms Oni2.Store.dispatch : After: (SetTitle "KanbanBoard.js - client - Onivim 2")
                                                                                [DEBUG]    +0ms Oni2.Store.StoreThread : Effect complete: Batch:
                                                                                scm.getOriginalUri
                                                                                indentation.detectAndSet
                                                                                title.update

                                                                                Yojson.Safe.Util.Type_error("Expected array, got null", 870828711)
                                                                                Raised at file "map.ml", line 135, characters 10-25
                                                                                Called from file "sexp_conv.ml", line 156, characters 10-37
                                                                                Uncaught Exception:  { [Error: read ECONNRESET
                                                                                at Pipe.onStreamRead (internal/stream_base_commons.js:111:27)] errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }

Looking at #1839 I guess this is a duplicate of it, but I guess its helpful to see that this issue exists on MacOS and Linux.

This looks to be because of the extension host side:
Unhandled Promise Rejection:  [Error: Could not create TS service
	at TypeScriptServiceClient.service (/Applications/Onivim2.app/Contents/Resources/extensions/typescript-language-features/out/typescriptServiceClient.js:188:15)
	at TypeScriptServiceClient.executeImpl (/Applications/Onivim2.app/Contents/Resources/extensions/typescript-language-features/out/typescriptServiceClient.js:481:41)
	at TypeScriptServiceClient.executeAsync (/Applications/Onivim2.app/Contents/Resources/extensions/typescript-language-features/out/typescriptServiceClient.js:473:21)
	at new GetErrRequest (/Applications/Onivim2.app/Contents/Resources/extensions/typescript-language-features/out/features/bufferSyncSupport.js:210:16)
	at Function.executeGetErrRequest (/Applications/Onivim2.app/Contents/Resources/extensions/typescript-language-features/out/features/bufferSyncSupport.js:221:16)
	at BufferSyncSupport.sendPendingDiagnostics (/Applications/Onivim2.app/Contents/Resources/extensions/typescript-language-features/out/features/bufferSyncSupport.js:379:63)
	at Delayer.diagnosticDelayer.trigger [as task] (/Applications/Onivim2.app/Contents/Resources/extensions/typescript-language-features/out/features/bufferSyncSupport.js:351:18)
	at completionPromise.Promise.then (/Applications/Onivim2.app/Contents/Resources/extensions/typescript-language-features/out/utils/async.js:26:50)]

which I'm assuming then causes the issue on the Reason side:
Yojson.Safe.Util.Type_error("Expected array, got null", 870828711)
Raised at file "map.ml", line 135, characters 10-25
Called from file "sexp_conv.ml", line 156, characters 10-37

Annoyingly it doesn't really tell us where that error is, at least not easily.
@bryphe, is this something that is known/is in progress? I see two issues, robustness for the Reason-side and actually getting the server started on the ext host side.

Thanks for the logging issue @Xatenev (and for the details @CrossR !)
Looks like it's a bug in deserialization of JSON from the extension host here: https://github.com/onivim/oni2/blob/5c362bf7b1766091bb70a31a16d9d0ba09bb63af/src/Exthost/Msg.re#L117
We're assuming that the $changeMany call is always a list of diagnostics, but seems like there is a case where it is null. Working on fixing the parsing + adding a test case now

This is fixed now with #1841 , so I'll close out. But please let me know if you're still able to reproduce.


Question: Unable to build Mvc.sln

Hello,
First of all sorry if this is not the right place to ask this kind of question, guide me if I'm wrong.
I have cloned the MVC repository form https://github.com/aspnet/Mvc.git, everything went well the dependent pacages were downloaded and all the stuff , the problem is when I try to rebuild the solution Mvs.sln I get the following errors:

Am I missing something ?
Thanks for help,
Regards,
Victor

You might have to go volatile at the moment e.g. add to the nuget.config in the root:
<add key="AspNetVolatile" value="https://www.myget.org/F/aspnetvolatiledev/api/v3/index.json" />
You also don't want to use it for long, as its more unreliable, so revert when you can...

Hi,
Thanks for help, passing to aspnet AspNetVolatile resolved all the build issues.
Regards,
Victor

Let me repeat this: DO NOT USE THE VOLATILE FEED. It it NOT for humans to use. It's strictly for making the CI machine to use. Using it will leave you broken 90% of the time. Our builds are unstable now because of the switch to dotnet CLI. It is coming back online quickly I'd recommend not building until then.

^----- what he says :)

^----- what he says - you only really want to use it for testing a "from source" compilation change; when the normal feed isn't working with the source. When the normal feed isn't working its because the tests are broken - which means you shouldn't use the volatile...

OK understood :) I reverted to the original feed and the solution builds without errors.


[FLINK-32684][rpc] Introduces RpcOptions and deprecates AkkaOptions

What is the purpose of the change
The name {{AkkaOptions}} is wrong after replacing Akka with Apache Pekko in FLINK-32468. The new options should be named more generically to avoid running into the same issue in the future when/if we decide to replace Pekko with gRPC (FLINK-29281).
Brief change log

Renames AkkaOptions into RpcOptions using the IDE
Copied RpcOptions to AkkaOptions, adds @Deprecated annotation to AkkaOptions and references corresponding RpcOptions in AkkaOptions.
Removes deprecated options from RpcOptions (but keeps them in AkkaOptions)

Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? not applicable

@flinkbot run azure

Sorry I missed the review request notification days ago. @XComp
I will take a look right away.


(Draft) Improve documentation

Description
This PR creates readme.md with details about image-generation and deployment in help folder.
Related issue:
https://github.com/actions/virtual-environments/issues/564
Check list

[ ] Related issue / work item is attached
[ ] Tests are written (if applicable)
[ ] Documentation is updated (if applicable)
[ ] Changes are tested and related VM images are successfully generated

TODO: Need more detailed info about linux image generation


Allow multiple snp_transcripts in plot_diplotype_clustering_advanced()

Resolves #600.
@KellyLBennett and Nana Amoako needed this functionality earlier today so I sped it up a bit. There are currently no tests for plot_diplotype_clustering_advanced() so the best I can say is that it worked in the notebook. I will add tests when I have time.

Here is a picture (the chosen genes were Cyp6aa1 and Cyp6p15p):

We might want to add titles to make things clearer.

awesome.


[v2.0.3] Moving utilities exceptions to their internals

#18  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @Haaabeeel and the rest of your teammates on  Graphite

Merge activity

Apr 23, 7:18 AM EDT: @Haaabeeel merged this pull request with Graphite.


resolve dependency conflict (tensorflow vs. numpy)

Tensorflow < 2.7.0 is incompatible with numpy 1.20.0 and newer.
Therefore I'm setting the numpy version to numpy~=1.19.2
(i.e. any version 1.19.x but at least 1.19.2) as this is the
version used in EQTransformerviaGoogleColab.ipynb.
Also set TF to tensorflow~=2.5.0 so that it can be used
with any 2.5.x version.
Also add requirements.txt with the same definitions, so that dependencies can be installed with:
pip install -r requirements.txt

Addresses: #99

Thank you @ostueker


Add in step for scoring during syn queue (but no email)

LGTM. Thanks @vpchung

@thomasyu888 Mike has requested the following for the Synthetic Validation phase:

run a submission model against the synthetic data, notifying the participant by email whether or not their model is valid (no scoring needed)
run that model again against the real data and score it, but participants should NOT be notified of these steps (scores are really just for us organizers)
logs should be saved onto Synapse for the synthetic data but not for the real data

I think the updates I made to the workflow have tackled these, but please let me know if I missed anything important!

@thomasyu888 Mike has requested the following for the Synthetic Validation phase:

run a submission model against the synthetic data, notifying the participant by email whether or not their model is valid (no scoring needed)
run that model again against the real data and score it, but participants should NOT be notified of these steps (scores are really just for us organizers)
logs should be saved onto Synapse for the synthetic data but not for the real data

I think the updates I made to the workflow have tackled these, but please let me know if I missed anything important!


Removed unused imports

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[x] Each commit in the pull request should have a meaningful subject line and body.
[ ] Format the pull request title like [JAV-XXX] Fixes bug in ApproximateQuantiles, where you replace JAV-XXX with the appropriate JIRA issue.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Run mvn clean install to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Fix https://github.com/ServiceComb/ServiceComb-Java-Chassis/issues/409

Changes Unknown when pulling b1e86490ff5434e69f1be5e2897b200becdbd90f on Gr1f0n6x:unused_imports into ** on ServiceComb:master**.