Recommended pattern to change/post data

Is there any recommended way to use useAsyncResource to change data in a way which is integrated with the suspense components?

useAsyncResoource works with any function that returns a Promise, which can be anything, including a POST request.
However, since after updating a piece of data you also need to retrieve it again, you would probably need to clear the cache and use the updater for your original GET request after the POST call is complete.
e.g.
// user data reader
const [user, getUser] = useAsyncResource(fetchUser, userId);

// user data updater
const userUpdaterFn = useCallback((newUserData) => {
  // api call to update the data
  updateUser(userId, newUserData)
    .then(()=> {
      // clear the cache
      resourceCache(fetchUser).delete(userId);
      // fetch the updated user data from the server
      getUser(userId);
    });
}, [userId]);

return (
  // ...
  <form onSubmit={userUpdaterFn} ... />
);

Alternatively, you can optimistically update the data in the cache if you know the exact format the server would return:
// user data reader
const [user, getUser] = useAsyncResource(fetchUser, userId);

// user data updater
const userUpdaterFn = useCallback((newUserData) => {
  // directly update the data in the cache (notice you must know the exact user object as returned by fetchUser)
  resourceCache(fetchUser).set(
    () =>  ({ userId, ...newUserData }), // a static data reader function
    userId, // the fetchUser function params - userId in this case
  );

  // refresh the user data reader (immediately re-render the componet with this new data)
  getUser(userId);

  // actually call the POST api to update the data on the server
  updateUser(userId, newUserData)
    .catch(() => /* only handle failures */);
}, [userId]);

Another option would be a mix of both, if the POST api call returns the same data as the GET one:
// user data updater
const userUpdaterFn = useCallback((newUserData) => {
  // api call to update the data
  updateUser(userId, newUserData)
    .then((updatedUserResponse)=> {
      // directly update the data in the cache with the new user object
      resourceCache(fetchUser).set(
        () =>  ({ userId, ... updatedUserResponse }),
        userId,
      );

      // refresh the user data reader (immediately re-render the componet)
      getUser(userId);
    });
}, [userId]);


Replaced all humbug_get_contents with the namespaced one

This PR fixes issue #45

Fixed, Travis build is succeeding so not sure what you mean with the prefer lowest failing?

Hm somehow it gets the 1.1.0 even if prefer lowest 🤔

Thanks! Will require the dev-master for now so at least I can finish my project :)

What about releasing a new version with these changes?

It will be released when I'll find the time to do a round of phar-updater, either require dev-master or that specific commit if that's urgent meanwhile

We can release a new version, I think, for this change. It will get is over the deprecation notice for current usage while the other more significant PRs are being reviewed.

I hope. Because currently it doesn't build out-of-the-box due to this notice.


GM: common longitudinal actuator delay

This helped on the Acadia (https://github.com/commaai/openpilot/pull/25354), as well as the new Bolt EV/EUV where both cars have a large delay to initially start braking (similar to lateral friction we must overcome). The Escalade merged here also seems to have this same delay: https://github.com/commaai/openpilot/pull/27276
This PR adds this delay to all camera ACC cars (EVs do not have regen capabilities), and all ICE cars. The only platform this does not affect is the Volt.

Our Bolt:

Waiting for a Silverado (camera ACC) and Escalade (ASCM) to try this and will merge.

Escalade: https://github.com/commaai/openpilot/pull/27406

Silverado also has similar lag: fe80a4e1cedec853|2023-03-20--20-24-23

Moving the Volt over as well, as the regen that is supposedly non-laggy isn't uniform across all speeds, and non-existent at low speeds, so the brakes are also reportedly laggy there.


Internationalize section labels in emoji picker

The section labels in the emoji picker are not internationalized. We should fix this.
See CZO thread for discussion, including technical considerations.

@zulipbot claim

Hello @alya for the emoji area please am I to use i18n to translate the different categories listed following the data structure path done and do them manually such that it refreshes directly once connected?
As seen from the emoji picker initialization.

@giftcup Please take a look at the Zulip contributor guide, and keep in mind the following guideline:

Before you claim an issue, you should be confident that you will be able to tackle it effectively.

I will go ahead and unassign this issue, and you should feel free to re-claim it once you have figured out how to approach it (or pick a different one if you prefer).
If you need help, you can post a question in #frontend on chat.zulip.org.


adjusting api call to new host - https://api.darksky.net

change of dns api.forcast.io to api.darksky.net
changing description slightly
adding new keywords

+1

Thanks @tobiasehlert!


Edited any file related to accessing the database

Made sure that they were accessing and/or modifying it and/or saving to it properly

Again, just added the commits to the previous request: edited all files pertaining to private event booking with the update versions of the entities and associated factories

thank you for updating the entities, it makes it helpful for me to build the app-builder.


dataset problem

Which habitat dataset version should I download? v0.1 or v0.2 ?  looking forward to your reply

Hi, we use v0.2 version.

Thank you very much for your reply. Could you please provide the specific code for downloading the data set? Is hm3d_minival_v0.2 enough? Or need other documents?

Hi, we downloaded the dataset from here. And you will also need val split for evaluation.

I have downloaded all the minival, train and val files for v0.2, but now there are only two folders, scene datasets and versioned data. Where can the objectgoal_hm3d folder and the other two files be downloaded?

Looks like you've already fixed the problem, closing the issue now.

Thank you very much for your reply. Could you please provide the specific code for downloading the data set? Is hm3d_minival_v0.2 enough? Or need other documents?

Could you tell me which package i should download in the HM3D dataset and how to prepare the data? Thanks


TestRulesAPI failed due to missing the object e2e_rules_api-rules-minio

The test TestRulesAPI failed while we using CI pipeline to containerize Observatorium-api and break the pipeline, the two sections of logs are related to this, may I ask your support regarding this?
11:01:20 rules-minio: Pull complete
11:01:20 rules-minio: Digest: sha256:a150b34a87e44bb8d9a862c22dd2a737e1918990c6a9f0513f9314c6270921ab
11:01:20 rules-minio: Status: Downloaded newer image for minio/minio:RELEASE.2021-07-27T02-40-15Z
11:01:20 opa: {"current_version":"0.31.0","download_opa":"[https://openpolicyagent.org/downloads/v0.37.1/opa_linux_amd64","latest_version":"0.37.1","level":"info","msg":"OPA](https://openpolicyagent.org/downloads/v0.37.1/opa_linux_amd64%22,%22latest_version%22:%220.37.1%22,%22level%22:%22info%22,%22msg%22:%22OPA) is out of date.","release_notes":"[https://github.com/open-policy-agent/opa/releases/tag/v0.37.1","time":"2022-02-03T11:01:20Z](https://github.com/open-policy-agent/opa/releases/tag/v0.37.1%22,%22time%22:%222022-02-03T11:01:20Z)"}
11:01:20 Ports for container e2e_logs_read_write_tail-opa >> Local ports: map[http:8181] Ports available from host: map[http:49190]
11:01:20 opa: {"client_addr":"<IP_ADDRESS>:38834","level":"info","msg":"Received request.","req_id":1,"req_method":"GET","req_path":"/health","time":"2022-02-03T11:01:20Z"}
11:01:20 opa: {"client_addr":"<IP_ADDRESS>:38834","level":"info","msg":"Sent response.","req_id":1,"req_method":"GET","req_path":"/health","resp_bytes":2,"resp_duration":3.09471,"resp_status":200,"time":"2022-02-03T11:01:20Z"}
11:01:20 Starting loki
11:01:20 loki: Unable to find image 'grafana/loki:2.3.0' locally
11:01:21 rules-minio: useradd: UID 0 is not unique

Error: No such object: e2e_rules_api-rules-minio

=== CONT  TestRulesAPI
    services.go:57: services.go:57:

         unexpected error: failed to get mapping for port as container e2e_rules_api-rules-minio exited: exit status 1: docker container rules-minio failed to start: exit status 1

11:01:52 Killing rules-minio
11:01:52 Error response from daemon: Cannot kill container: e2e_rules_api-rules-minio: No such container: e2e_rules_api-rules-minio

11:01:52 Unable to kill service rules-minio : exit status 1
11:01:52 Killing opa
11:01:53 Killing gubernator
11:01:53 Killing dex
--- FAIL: TestRulesAPI (53.75s)
FAIL
FAIL	[github.com/observatorium/api/test/e2e](http://github.com/observatorium/api/test/e2e)	95.152s
FAIL
Makefile:93: recipe for target 'test-e2e' failed
make: *** [test-e2e] Error 1

Hi @Xxiao-ma , thanks for reporting this :)
Can you also reproduce this problem locally? (by running make test-e2e)
I think this can be related to how the minio server is instantiated - using NewMinio here which leads to here
There is an open issue but I'm not sure if its 100% related to that. @matej-g did you happen to encounter the same problem?

Hey @Xxiao-ma @jessicalins,
To me this seems to not be directly related to https://github.com/efficientgo/e2e/issues/11, but could have something to do with how the image is being used, where a new user is created as part of the command (see https://github.com/efficientgo/e2e/blob/main/db/db.go#L72).
Do you have any more details @Xxiao-ma, what kind of CI are you running in? Are you perhaps running is as a root user and that could be causing the error you see regarding non-unique UID?

Hi @jessicalins @matej-g , thanks for replying
I'm using the Tekton Pipelines , more details can be found here. I also guess the issue is as @matej-g explained due to the user creation. I'm not sure if I can change the config of how to run the CI Pipeline (as a non root user), since the Tekton we use is SaaS.

Hm, can you perhaps adjust the testing code on your fork, maybe use different approach with Minio, since we currently depend on the workaround I linked before, so I'm not sure if there's an easy way around it.


Why is Safari

I DO NOT KNOW IF THIS IS A GOOD IDEA, SOMEONE PLEASE CHECK IF I'M LOSING MY MIND
Sorry about the length of this description, feel free to just read the code and come back to this if you wonder "But why?"
"But why?"
Because Safari
Today when testing out some audio issues in another demo, @jamsea and I tried hopping into CoD. We noticed that media tracks are not auto played despite active gestures on the screen being made in Safari, until the user turned on their own camera or microphone. In the screenshot below the "chrome" user is actually unmuted (both camera and mic). "You" (the safari user) do not see that chrome user until you unmute one of your own devices.

You should be able to easily repro this by going to the deployed version of CoD, at least on Mac OS Monterey 12.6 (prepare for a slow initial loading time due to some render.com throttling): http://code-of-daily-modern-wordfare.onrender.com :

Open the above in Chrome. Turn on video and audio
Click the invite link in the controls down below and paste the URL in Safari
Note that the Chrome participant's media is not played.
Turn on your own camera
Note that both participants' media is now played

NotAllowedError and autoplay
This seems to be an autoplay permission error, in that despite the user making "gestures" on the page, the unmuted video is still not allowed to play. This plays fine in Chrome and FF (in my testing), but Safari is Safari.
Remediation 1: scrap autoplay
So in this PR I am trying to scrap autoplay and call play() on the video tag in the following cases:

A new MediaStream is added as a video srcObject
Tracks on an existing MediaStream are replaced

Remediation 2: an added flow to handle NotAllowedError
Originally, I was getting the same permission error in Safari even with the above technique change. The above was added just so that we could catch the error and respond. To that end, the PR adds a new error: ErrMediaAutoplay, which is shown when this particular permission error is hit. It encourages the user to hit a given "Play" button manually, which Safari seems to be happy with as a user gesture to begin playing. The error then gathers up all participant video elements in existence in the DOM and calls play() on them (a brute-force attempt to play all the things)
BUT WAIT, THERE'S MORE
The above worked well in my testing, until my entire machine froze up and I had to reboot, which also included a Safari update. AT THIS POINT, just the technique change of removing autoplay and calling play() instead (as described in "Remediation 1" above) seems to avoid the permission error. So the error message described is no longer consistently triggered for me.
But considering we've seen so much wonkiness with Safari at this point, I propose leaving this in, in case another Safari version or some other configuration does still trigger the permissions error. This way, whoever runs into this has a way to (hopefully) fix it and keep playing.

(Aside: if this does end up being the correct approach we probably want to say something about this in our upcoming track handling post...)

(Aside 2: This PR does not address the following video issue I posted about earlier, also seemingly triggered by Safari)

This seems to be an autoplay permission error, in that despite the user making "gestures" on the page, the unmuted video is still not allowed to play. This plays fine in Chrome and FF (in my testing), but Safari is Safari.

The important piece here is that not all "gestures" are "gestures to allow autoplay", i.e. a click on a button that happened before a video was rendered, doesn't count as a gesture to allow autoplay. You will want to play audio only from resolved getUserMedia promises or user event callbacks. It doesn't really matter if failing to play media originates from the autoplay attribute or a programmatic call to play(). Resolving a failed autoplay is a little more difficult to handle, as it's less predictable when a media element attempts to autoplay.
In Prebuilt we're utilizing the canplay and play DOM events to detect failed autoplay attempts. We'll assign a timeout of 1500ms inside the canplay handler and clear that timeout inside the play handler. In case there's no play event, we'll show some UI to ask for a user interaction ("Please click this button") and inside that click handler we'll attempt to play() all paused media elements.


How to disable CFG in win10

Recently you add some new code here:https://github.com/Microsoft/ChakraCore/commit/c0c3de7acdc167ca93b689f7552a5a0fe4bc8be2#diff-368bbd31c6049b4faa033bbbc971ef0f
But there are some old win10 versions that does not support cfg(my test enviroment is 10.0.14393).
And when I build ch.exe, there is a fatal error that "no such file cfguard.h" ...etc

@LouisLaf

Hi! Just want to to add to the report that eshost builds are failing on appveyor with the same error message.

Yes, it looks like the VS 2015 builds are broken.  CFGuard.h was added in VS 2017.  I'll look into it.

VS2015 Update #3 should build fine now.


crash with some fb videos url

Hello
i tried with some fb video url for eg:
https://www.facebook.com/streetfoodindiaofficial/videos/bhang-wali-thandaibhang-thandai-varanasi-reelindia-exploreindia/651222133430822
https://www.facebook.com/oil.oil.6969/videos/1104893156878184/
and pm2 got restart with following error
2023-02-24T13:58:36: TypeError [Error]: Cannot read properties of null (reading '1')
2023-02-24T13:58:36:     at /www/wwwroot/appi/node_modules/fb-downloader/index.js:41:69
2023-02-24T13:58:36:     at processTicksAndRejections (node:internal/process/task_queues:96:5)
Please check .Thank you

probably the cookies issue, change the cookies in the header to your fb account one.


Storage bootstrap error

Finished dev [unoptimized + debuginfo] target(s) in 0.57s
Running `target/debug/risinglight -f tests/sql/basic_test.slt`
2022-03-07T12:37:52.264327Z  INFO risinglight: using Secondary engine
2022-03-07T12:37:52.269119Z  INFO risinglight::storage::secondary::storage: applying 9 manifest entries
2022-03-07T12:37:52.269753Z  INFO risinglight::storage::secondary::storage: 0 tables loaded, 3 rowset loaded, 0 DV loaded
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', /Users/hwy/Code/Rust/risinglight/src/storage/secondary/storage.rs:131:53
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

This error occurs when cargo run -- -f tests/sql/basic_test.slt is run the second time
The code snippet for specifying the bug looks like this
// risinglight/src/storage/secondary/storage.rs:131
  for (_, entry) in rowsets_to_open {
            let table = tables.get(&entry.table_id).unwrap(); // error panic
            let disk_rowset = DiskRowset::open(
                table.get_rowset_path(entry.rowset_id),
                table.columns.clone(),
                engine.block_cache.clone(),
                entry.rowset_id,
                options.io_backend,
            )
            .await?;
            changeset.push(EpochOp::AddRowSet((entry, disk_rowset)));
        }

There are no tables after reading the manifest file, but there are rowsets, I tried to use match to handle None, and the error didn't recur.
If this is a bug, can I submit a PR to fix it? (I am a rust novice

The underlying cause behind this panic is because that we didn't support drop table. Here I think it's okay to simply continue the for loop if the table doesn't exist. Welcome to send a PR!

I've also created a tracking issue for supporting drop table. If you are interested in fixing the root cause (instead of work-around), you may also start working on this issue https://github.com/risinglightdb/risinglight/issues/553. Thanks!

Yeah, I'd love to. Thanks for answering so quickly(God Chi !)


NCD - Next Steps, diagnostics logic

#404
#405
#406
#407
#408
#409
#410

@AronNovak please review


Coordinated releases with algaeff 2.0.0

@jonsterling proposed a plan to coordinate releases in #24 in the face of the next major release with breaking changes. Let's continue the conversation here.

asai: https://github.com/RedPRL/asai/pull/132
yuujinchou: https://github.com/RedPRL/yuujinchou/pull/123
forester: ?

@jonsterling algaeff 2.0.0 and yuujinchou 5.2.0 are ready! So potentially you could start updating forester now...

Thanks @favonia ! I'm cutting a new version of forester that adds the upper bounds, and then once your new packages appear on opam I will upgrade forester.

@jonsterling I thought you can just update the existing OPAM files in the OPAM repository and maybe that would be easier (?)

@jonsterling Actually, I have perhaps a better plan: I have figured out how to release multiple packages at once. I can release all four OPAM packages when you are ready.

Cool :) If you can wait until my PR is merged (https://github.com/ocaml/opam-repository/pull/24735), then I will be very happy

Now the coast in clear but I want to release everything with asai 0.3.0 altogether!

Done by https://github.com/ocaml/opam-repository/pull/24738.


checkov: 2.0.641 -> 2.0.648

Motivation for this change
Update to latest upstream release 2.0.648
Change log: https://github.com/bridgecrewio/checkov/releases/tag/2.0.648
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 149913 run on x86_64-linux 1

1 package built:

checkov


Is resend action in JdbcTransactionalStorage non-transacted

❗ please do not add sensitive information in issues, you can provide extra information via email using issue number as reference ❗
Describe the issue
I had a few messages in error store. I resent the messages. All failed to be processed at some point in the pipeline. But the messages are never returned to the error store.
Below I paste the receiver and the pipeline. The pipeline throws an error when resolving the ServiceCode param since the messageContext is not stored in the error store the contextKey is not available and converting NULL to number fails. However, the message in error store are not restored.
<receiver numThreads="4" transactionAttribute="Required">
			<listener name="FF_Common_1" destinationName="dest.queue" lookupDestination="false" queueConnectionFactoryName="jms/qcf" messageProtocol="FF" className="nl.nn.adapterframework.extensions.esb.EsbJmsListener"/>
			<errorStorage slotId="/FF_Common_1" className="nl.nn.adapterframework.jdbc.JdbcTransactionalStorage"/>
		</receiver>
		<pipeline>
			<exit name="EXIT" state="SUCCESS"/>
			<exit name="ERROR" state="ERROR"/>
			<pipe name="logStart" className="nl.nn.adapterframework.pipes.LogContextPipe">
				<param name="ibis" value="und"/>
				<param name="service" value="Configuration_FF_Common_1"/>
				<param name="service_type" value="SOAP"/>
				<param name="conversationId" sessionKey="soapHeader" xpathExpression="//*:ConversationId" defaultValue="notfound"/>
			</pipe>
			<pipe name="Call StoredProcedure" storeResultInSessionKey="output" className="nl.nn.adapterframework.pipes.SenderPipe">
				<sender datasourceName="jdbc/oracle" timeout="30" query="{ ? = call test(?, ?, ?) }" queryType="other" className="nl.nn.adapterframework.jdbc.StoredProcedureQuerySender">
					<param name="output" mode="OUTPUT" type="CHARACTER"/>
					<param name="ServiceCode" sessionKey="originalMessage" contextKey="ServiceCode" type="NUMBER"/>
					<param name="Content" type="CHARACTER"/>
					<param name="P_stub" value="FALSE"/>
				</sender>
				<forward name="timeout" path="TimeoutError"/>
			</pipe>

Reporter
Ali/Jeroen
FF: 7.9-20230811.190015

Can you try with transactionAttribute="Required" on pipeline?

Receiver#retryMessage is transactional?

We should rename this issue to "When messages are resend from the errorstore and run into error they disappear"
It could be the fact that we are using a non XA database connection and the adapter is transacted and connects to JMS?
How it could work better!

Message is picked up from the errorstore (dbtable) status is set to in progress
Message is put on the adapter
If an exception occurs:

The error result is updated in the database
Resend history is added (user | dateTime | number of try)
The message is kept for resending
if the message is processed without errors
the status is updated to succes
expiry date is set

Created issue #6393

At this time the errorstore needs to use a transactional datasource for the errorstore to work correctly. This is tested and has the required behavior.
As a feature it would be nice to have the setup as described earlier in this issue so we can use similar solution as used for the error store on filesystem. Add a hook to resend via a schedule and even have a resend history.

We can close the issue and I will create a feature request


Need to update CI to new version

https://github.com/FirewallRobotics/2024-Zenith/blob/4e2da481071fd200309edfc88d70093131aa575a/.github/workflows/main.yml#L21C50-L21C50
2023-22.04 needs to be updated to 2024-22.04.
For reference -
https://docs.wpilib.org/en/stable/docs/software/advanced-gradlerio/robot-code-ci.html

She did this 2 weeks ago and forgot to close :)


Chore: Seperate Hetzner from tests-sets

Description

We've alot of issues with hetzner when the e2e pipeline receives IP's that were misused in some form. To name a few, docker images may fail to download on the build cluster, IP may be banned from certain endpoints that are part of the building workflow and thus the workflow fails etc...
Seperate hetzner from the test-sets into its own "job",  so that if we receive such IP in our e2e pipeline we can still determine if the changes made were breaking or not by looking at the test-set that didn't include hetzner
Exit criteria

[ ] Seperate Hetzner (possibly by including a seperate test-set or creating a new stage in the e2e pipeline)

We agreed that we should reduce the uses of hetzner among the test-cases


lifted.Tree.freeVariables distinguishes Variables by type

The function freeVariables distinguishes Variables that have the same symbol but different type:
https://github.com/effekt-lang/effekt/blob/8365cbd7f24cd1a59f7c9b5a7a21a1c4dc2c7826/effekt/shared/src/main/scala/effekt/lifted/Tree.scala#L112-L177
This is a problem:

When types are syntactically different (e.g. alpha-equivalence, capabilities changing along the enclosing scopes)

If we want to prevent this, we'd have to rewrite the types in appropriate positions

When types are computed incorrectly

2 occurs in PolymorphismBoxing e.g. in the following program:
effect Increment {
  def increment(): Unit
}
def main() = {
  try {

    def step(): Int = {
        def looper(): Int / Increment = 0
        looper()
    }

    step()
  } with Increment {
      def increment() = resume(())
  }
}

because the types in argument position BlockVars are not transformed correctly.

Yes, we should fix both, then.


Weird issue with CRLF on submodule

Preliminary Steps
Please confirm you have...

[x] reviewed How Linguist Works,
[x] reviewed the Troubleshooting docs,
[] considered implementing an override,
[x] verified an issue has not already been logged for your issue (linguist issues).

Problem Description
I'm seeing a bizarre behavior with the Typescript-TmLanguage submodule - it shows as having a different change in the working tree. Nothing I have done has helped, including these commands:
git submodule foreach --recursive git clean -xfd
git reset --hard
git submodule foreach --recursive git reset --hard
git submodule update --init --recursive

It seems to have something to do with CRLF. I have not changed any git settings. I'm using OSX 10.14.
|kbrown ~/dev/github/linguist/vendor/grammars/TypeScript-TmLanguage[(HEAD detached at dc5fe35) !]
| > cat .gitattributes
# Make sure baseline files have consistent line endings
*.txt text eol=lf

This is what I see what I try to see status:
|kbrown ~/dev/github/linguist/vendor/grammars[plantuml !]
| > git st
On branch plantuml
Your branch is up to date with 'origin/plantuml'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  (commit or discard the untracked or modified content in submodules)
	modified:   TypeScript-TmLanguage (modified content)

no changes added to commit (use "git add" and/or "git commit -a")
|kbrown ~/dev/github/linguist/vendor/grammars[plantuml !]
| > git diff
diff --git a/vendor/grammars/TypeScript-TmLanguage b/vendor/grammars/TypeScript-TmLanguage
--- a/vendor/grammars/TypeScript-TmLanguage
+++ b/vendor/grammars/TypeScript-TmLanguage
@@ -1 +1 @@
-Subproject commit dc5fe357ee5582244b798917a5fc9d7cd73e6220
+Subproject commit dc5fe357ee5582244b798917a5fc9d7cd73e6220-dirty
|kbrown ~/dev/github/linguist/vendor/grammars[plantuml !]
| > cd TypeScript-TmLanguage/
|kbrown ~/dev/github/linguist/vendor/grammars/TypeScript-TmLanguage[(HEAD detached at dc5fe35) !]
| > git st
HEAD detached at dc5fe35
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   ThirdPartyNotices.txt

no changes added to commit (use "git add" and/or "git commit -a")
|kbrown ~/dev/github/linguist/vendor/grammars/TypeScript-TmLanguage[(HEAD detached at dc5fe35) !]
| > git difftool
warning: CRLF will be replaced by LF in ThirdPartyNotices.txt.
The file will have its original line endings in your working directory

Viewing (1/1): 'ThirdPartyNotices.txt'
Launch 'bc3' [Y/n]? n

URL of the affected repository:

This doesn't sound like a Linguist-specific issue but rather a general git/user-environment or tools issue.
I suspect something (maybe your editor, an extension in your editor, or even the difftool you're using) may be iterating over all the submodules and performing some action (maybe git status?) which in turn is causing the .gitattributes to be assessed.
To troubleshoot this, I'd suggest shutting down all editors and tools that may touch repos, remove all modifications to the global git config file, remove all local and user git config files and then clone the repo from the CLI.  A quick way to do this would be to create a new user without any customisations to their env.
This should work without any problems and standard git commands shouldn't report any changes (they don't in my env). You can then slowly add back your customisations to work out which one is triggering this behaviour.

I'm still not sure what caused this, but it was fixed by following this answer:
https://stackoverflow.com/a/27415757/4907881
specifically:
git submodule deinit -f .
git submodule update --init


Switch from per chromosome to autosomes

Hello Dr. Atkinson,
Thank you so much for these scripts and example code.
In the RFMix_v2 step, your example code produces per chromosome rfmix output.
In the Extract Tracts step, it's unclear whether you've used per chromosome or an autosome file (I used per chromosome MSP files and VCF files).
However in the example Hail code you've read in an autosomes.anc0.dosage.txt file.
I'm wondering where, when, and how the autosomes file should be made. Or rather, did you run RFMix_v2 on the whole genome instead of by chromosome as the example suggests?
Thank you for clarifying!
Heidi

Still having the same issue. I noticed the previous issue (#6 ) was solved by harmonizing the msp and vcf files. Do the msp and vcf files need to be by chromosome or all autosomes?

Hi Heidi,
Apologies I missed this. To make the autosomes file I simply concatenated all the individual chromosome files output by RFmix together prior to running the Tractor scripts. This way I only needed to launch one job to run tractor GWAS across all the autosomes. You could still run things on a chromosome by chromosome basis, however if that is preferable - both options should work now as the code has been updated.
Let me know if this works!
Best,
Elizabeth


Non Resourceful URLs

Hi, i'm using a RoR back-end and have many arbitrary URLs for some actions, e.g:
POST /posts/:id/confirmation
At the moment, i'm overriding resourceRelationshipURL with something like this
const relationshipPath = (relationship === 'confirmation') ? '/' : '/relationships/';
 return (
   this.resourceURL(type, id) +
   relationshipPath +
   this.serializer.resourceRelationship(type, relationship)
);

// The way i make the request
PostsRemoteStore.update(q => q.addToRelatedRecords(
  { id: '', type: 'post' }, 'confirmation',
  { id: `${post.id}`, type: 'post' }
))

Is there a better way to do this ? I understand that It's not JSON:API compliant, but sometimes custom URLs are needed.

Same issue. Any custom business logic needs non resourceful URLs. For example, if you are building twitter feed. There'll be a /feed URL which is different from /tweets URL, where both server same type of resources and feeds is custom to each user's tweet feed. It does not conform to typical JSON:API structure where a resource type has a single endpoint, any real-life API rarely does.
How do we model this using Orbit? We may have /feeds/, '/feeds/nearby, '/feeds/trending/ and many other URLs serving same resource type. It is possible in GraphQL to have many queries return the same type of data, feed, nearbyFeed, trendingFeed, etc

This is possible by specifying a url on your request options. https://github.com/orbitjs/orbit/issues/878#issuecomment-892302834


unable to connect smtp server with correct username and password

Hello,
I was trying to enable alerts for my environment. This is my configuration
[root@cvlogs elastalert]# cat config.yaml
rules_folder: example_rules
run_every:
minutes: 1
buffer_time:
minutes: 15
es_host: <IP_ADDRESS>
es_port: 9200
writeback_index: elastalert_status
alert_time_limit:
days: 2
smtp_host: 'smtp.gmail.com'
smtp_port: 465
smtp_ssl: true
from_addr: 'myemailid'
smtp_auth_file: '/opt/elastalert/smtp_auth_file.yaml'
================
[root@cvlogs elastalert]# cat /opt/elastalert/smtp_auth_file.yaml
user: emailid
password: password.
When I trying to test, getting below error.
ERROR:root:Error while running alert email: Error connecting to SMTP host: (535, '5.7.8 Username and Password not accepted. Learn more at\n5.7.8  https://support.google.com/mail/?p=BadCredentials q5sm33621602pgf.45 - gsmtp')
ERROR:root:Error while running alert email: Error connecting to SMTP host: (535, '5.7.8 Username and Password not accepted. Learn more at\n5.7.8  https://support.google.com/mail/?p=BadCredentials p68sm109068447pfd.11 - gsmtp')
ERROR:root:Error while running alert email: Error connecting to SMTP host: (535, '5.7.8 Username and Password not accepted. Learn more at\n5.7.8  https://support.google.com/mail/?p=BadCredentials w11sm109054090pfk.75 - gsmtp')
ERROR:root:Error while running alert email: Error connecting to SMTP host: (535, '5.7.8 Username and Password not accepted. Learn more at\n5.7.8  https://support.google.com/mail/?p=BadCredentials i124sm109962921pgd.15 - gsmtp')
ERROR:root:Error while running alert email: Error connecting to SMTP host: (535, '5.7.8 Username and Password not accepted. Learn more at\n5.7.8  https://support.google.com/mail/?p=BadCredentials 135sm109855541pgc.7 - gsmtp')
ERROR:root:Error while running alert email: Error connecting to SMTP host: (535, '5.7.8 Username and Password not accepted. Learn more at\n5.7.8  https://support.google.com/mail/?p=BadCredentials e1sm94182379pga.37 - gsmtp')
INFO:elastalert:Queried rule E2live Rule from 2016-12-30 05:56 UTC to 2016-12-30 06:11 UTC: 215 / 215 hits
INFO:elastalert:Ran E2live Rule from 2016-12-30 05:56 UTC to 2016-12-30 06:11 UTC: 215 query hits, 0 matches, 0 alerts sent
INFO:elastalert:Sleeping for 56 seconds
^CINFO:elastalert:SIGINT received, stopping ElastAlert..
Please let me know how to fix this.

Try changing your smtp_auth_file: '/opt/elastalert/smtp_auth_file.yaml' to smtp_auth_file: /opt/elastalert/smtp_auth_file.yml

You should also do this. http://serverfault.com/questions/635139/how-to-fix-send-mail-authorization-failed-534-5-7-14. This will allow elastalert to send mails from gmail.

Thanks Reeenz20 for your suggestions but there is no luck, I have already enabled those things.

Have you tried removing single quotes from your rule? Also you can try using TLS instead of SSL?

removed single quotes still no luck

Is your password working when you log it on using gmail? Try changing your password with alphanumeric only and remove special characters?

there is no special chars in my password and it is working perfect

Have you tried a different email account? Also can you paste your config file again?

I used two email addresses, both doesnt have any special char in password.
here is my config.
This is the folder that contains the rule yaml files
Any .yaml file will be loaded as a rule
rules_folder: example_rules
How often ElastAlert will query Elasticsearch
The unit can be anything from weeks to seconds
run_every:
minutes: 1
ElastAlert will buffer results from the most recent
period of time, in case some log sources are not in real time
buffer_time:
minutes: 15
The Elasticsearch hostname for metadata writeback
Note that every rule can have its own Elasticsearch host
es_host: <IP_ADDRESS>
The Elasticsearch port
es_port: 9200
Optional URL prefix for Elasticsearch
#es_url_prefix: elasticsearch
Connect with TLS to Elasticsearch
#use_ssl: True
Verify TLS certificates
#verify_certs: True
GET request with body is the default option for Elasticsearch.
If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
for details
#es_send_get_body_as: GET
Option basic-auth username and password for Elasticsearch
#es_username: someusername
#es_password: somepassword
The index on es_host which is used for metadata storage
This can be a unmapped index, but it is recommended that you run
elastalert-create-index to set a mapping
writeback_index: elastalert_status
If an alert fails for some reason, ElastAlert will retry
sending the alert until this time period has elapsed
alert_time_limit:
days: 2
smtp_host: smtp.gmail.com
smtp_port: 465
smtp_ssl : true
from_addr<EMAIL_ADDRESS>smtp_auth_file: /opt/elastalert/smtp_auth_file.yml

Try commenting these lines:
highlight=send_get_body_as#transport
smtp_host: smtp.gmail.com
smtp_port: 465
smtp_ssl : true
from_addr<EMAIL_ADDRESS>smtp_auth_file: /opt/elastalert/smtp_auth_file.yml
then move these lines to your rules file:
smtp_host: smtp.gmail.com
smtp_port: 465
smtp_ssl : true
from_addr<EMAIL_ADDRESS>smtp_auth_file: /opt/elastalert/smtp_auth_file.yml

same error..
[root@cvlogs elastalert]# python -m elastalert.elastalert --verbose --rule example_rules/my_execeptions.yaml
INFO:elastalert:Starting up
INFO:elastalert:Queried rule My Rule from 2017-01-03 11:57 UTC to 2017-01-03 12:12 UTC: 235 / 235 hits
ERROR:root:Error while running alert email: Error connecting to SMTP host: (535, '5.7.8 Username and Password not accepted. Learn more at\n5.7.8  https://support.google.com/mail/?p=BadCredentials a66sm137503934pfa.64 - gsmtp')
INFO:elastalert:Ran My Rule from 2017-01-03 11:57 UTC to 2017-01-03 12:12 UTC: 235 query hits, 1 matches, 0 alerts sent
INFO:elastalert:Sleeping for 59 seconds
^CINFO:elastalert:SIGINT received, stopping ElastAlert...
[root@cvlogs elastalert]#

/opt/elastalert/smtp_auth_file.yml file should have these lines:
user<EMAIL_ADDRESS>password: yourpassword
Please check your iptable if port 465 for output is allowed

[root@cvlogs elastalert]# cat /opt/elastalert/smtp_auth_file.yml
user<EMAIL_ADDRESS>password: mypassword
[root@cvlogs elastalert]#
able to telnet the port 465.
[root@cvlogs elastalert]# telnet smtp.gmail.com 465
Trying <IP_ADDRESS>...
Connected to smtp.gmail.com.
Escape character is '^]'.

Can you paste the whole example_rules/my_execeptions.yaml

es_host: <IP_ADDRESS>
name: My Rule
type: frequency
index: filebeat-*
num_events: 50
timeframe:
hours: 4
filter:

term:
message: "584c319a3950237600133ad7"

alert:

"email"

email:

<EMAIL_ADDRESS>smtp_host: smtp.gmail.com
smtp_port: 465
smtp_ssl : true
from_addr<EMAIL_ADDRESS>smtp_auth_file: /opt/elastalert/smtp_auth_file.yml

Can you place these lines after message: "584c319a3950237600133ad7" and it should be the same level as filter (not idented):
smtp_host: smtp.gmail.com
smtp_port: 465
smtp_ssl : true
from_addr<EMAIL_ADDRESS>smtp_auth_file: /opt/elastalert/smtp_auth_file.yml

tried same still same issue, if you dont mind if you have any working rule, please share with me.

Alert when the rate of events exceeds a threshold
(Optional)
Elasticsearch host
es_host: elasticsearch.example.com
(Optional)
Elasticsearch port
es_port: 14900
(OptionaL) Connect with SSL to Elasticsearch
#use_ssl: True
(Optional) basic-auth username and password for Elasticsearch
#es_username: someusername
#es_password: somepassword
(Required)
Rule name, must be unique
name: Filesystem Error Rule
(Required)
Type of alert.
the frequency rule type alerts when num_events events occur with timeframe time
type: frequency
(Required)
Index to search, wildcard supported
index: topbeat-*
(Required, frequency specific)
Alert when this many documents matching the query occur within a timeframe
num_events: 1
(Required, frequency specific)
num_events must occur within this amount of time to trigger an alert
timeframe:
minutes: 1
aggregation:
minutes: 1
aggregation_key: 'host'
realert:
minutes: 0
(Required)
A list of Elasticsearch filters used for find events
These filters are joined with AND and nested in a filtered query
For more info: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html
filter:
#- query:
query_string:
query: "message: php error"

query:
match_phrase:
fs.device_name: "/dev/simfs"

range:
fs.used_p:
from: 0.00
to: 0.01

(Required)
The alert is use when a match is found
alert:

"email"

alert_text_type: alert_text_only
alert_text: |
Filesystem Critical.
Host: {0}
Filesystem Available: {2:x} GB
Filesystem Used: {3}/1073741824
Device Name: {4}
Filesystem Percent Used: {5:.2%}
alert_text_args:

host
"@timestamp"
fs.avail
fs.used
fs.device_name
fs.used_p

smtp_host: mysmtpserver
smtp_port: 465
smtp_ssl: true
smtp_auth_file: /root/elastalert/auth_file.yml
from_addr<EMAIL_ADDRESS>(required, email specific)
a list of email addresses to send alerts to
email:

<EMAIL_ADDRESS>

thanks, will try and let you know!

Okay! That only works if you have topbeat installed in your clientserver. If not, just replace it with logstash or filebeat.

I used the difference mail address, now it is working perfect. Thanks for your support.
I changed only mail address.

Hi @reeenz20
I am getting following error when I include following lines at below of the example_frequency.yaml :
Traceback (most recent call last):
File "/usr/local/bin/elastalert-test-rule", line 11, in
load_entry_point('elastalert==0.1.12', 'console_scripts', 'elastalert-test-rule')()
File "/usr/local/lib/python2.7/dist-packages/elastalert-0.1.12-py2.7.egg/elastalert/test_rule.py", line 358, in main
test_instance.run_rule_test()
File "/usr/local/lib/python2.7/dist-packages/elastalert-0.1.12-py2.7.egg/elastalert/test_rule.py", line 337, in run_rule_test
rule_yaml = load_rule_yaml(args.file)
File "/usr/local/lib/python2.7/dist-packages/elastalert-0.1.12-py2.7.egg/elastalert/config.py", line 118, in load_rule_yaml
loaded = yaml_loader(filename)
File "build/bdist.linux-x86_64/egg/staticconf/loader.py", line 168, in yaml_loader
File "build/bdist.linux-x86_64/egg/yaml/init.py", line 71, in load
File "build/bdist.linux-x86_64/egg/yaml/constructor.py", line 37, in get_single_data
File "build/bdist.linux-x86_64/egg/yaml/composer.py", line 36, in get_single_node
File "build/bdist.linux-x86_64/egg/yaml/composer.py", line 55, in compose_document
File "build/bdist.linux-x86_64/egg/yaml/composer.py", line 84, in compose_node
File "build/bdist.linux-x86_64/egg/yaml/composer.py", line 127, in compose_mapping_node
File "build/bdist.linux-x86_64/egg/yaml/parser.py", line 98, in check_event
File "build/bdist.linux-x86_64/egg/yaml/parser.py", line 439, in parse_block_mapping_key
yaml.parser.ParserError: while parsing a block mapping
in "example_rules/example_frequency.yaml", line 20, column 1
expected , but found ''
in "example_rules/example_frequency.yaml", line 57, column 3
Can you please suggest what I am doing wrong.
Thanks

Did you edit anything on example_frequency.yaml? Can you paste this rule example_frequency.yaml?

Here it is  :
Alert when the rate of events exceeds a threshold
(Optional)
Elasticsearch host
es_host: watchmandev
(Optional)
Elasticsearch port
es_port: 9200
(OptionaL) Connect with SSL to Elasticsearch
#use_ssl: True
(Optional) basic-auth username and password for Elasticsearch
#es_username: anjans
#es_password: Anjan@123
(Required)
Rule name, must be unique
name: Example frequency rule
(Required)
Type of alert.
the frequency rule type alerts when num_events events occur with timeframe time
type: frequency
(Required)
Index to search, wildcard supported
index: logstash-*
(Required, frequency specific)
Alert when this many documents matching the query occur within a timeframe
num_events: 2
(Required, frequency specific)
num_events must occur within this amount of time to trigger an alert
timeframe:
hours: 4
(Required)
A list of Elasticsearch filters used for find events
These filters are joined with AND and nested in a filtered query
For more info: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html
filter:

term:
host.keyword: "ERROR"

(Required)
The alert is use when a match is found
alert:

"email"

(required, email specific)
a list of email addresses to send alerts to
email:

<EMAIL_ADDRESS>smtp_host: ExternalSMTPRelay
smtp_port: 465
smtp_ssl : true
from_addr<EMAIL_ADDRESS>smtp_auth_file: /opt/elastalert/smtp_auth_file.yml

try putting these lines
smtp_host: ExternalSMTPRelay
smtp_port: 465
smtp_ssl : true
from_addr<EMAIL_ADDRESS>smtp_auth_file: /opt/elastalert/smtp_auth_file.yml
above email:

Hi @reeenz20
Still some syntax error is coming like :
Traceback (most recent call last):
File "/usr/local/bin/elastalert-test-rule", line 11, in
load_entry_point('elastalert==0.1.12', 'console_scripts', 'elastalert-test-rule')()
File "/usr/local/lib/python2.7/dist-packages/elastalert-0.1.12-py2.7.egg/elastalert/test_rule.py", line 358, in main
test_instance.run_rule_test()
File "/usr/local/lib/python2.7/dist-packages/elastalert-0.1.12-py2.7.egg/elastalert/test_rule.py", line 339, in run_rule_test
conf = self.load_conf(rule_yaml, args)
File "/usr/local/lib/python2.7/dist-packages/elastalert-0.1.12-py2.7.egg/elastalert/test_rule.py", line 301, in load_conf
load_options(rules, conf, args.file)
File "/usr/local/lib/python2.7/dist-packages/elastalert-0.1.12-py2.7.egg/elastalert/config.py", line 148, in load_options
raise EAException("Invalid Rule file: %s\n%s" % (filename, e))
elastalert.util.EAException: Invalid Rule file: example_rules/example_frequency.yaml
{'smtp_host': 'ExternalSMTPRelay', 'smtp_port': 465, 'smtp_auth_file': '/opt/elastalert/smtp_auth_file.yml', 'smtp_ssl': True, 'from_addr'<EMAIL_ADDRESS>is not of type 'string', 'array'
Failed validating 'type' in schema['properties']['email']:
{'items': {'type': 'string'}, 'type': ['string', 'array']}
On instance['email']:
{'from_addr'<EMAIL_ADDRESS>'smtp_auth_file': '/opt/elastalert/smtp_auth_file.yml',
'smtp_host': 'ExternalSMTPRelay',
'smtp_port': 465,
'smtp_ssl': True}
Do we need to include double quotes like -<EMAIL_ADDRESS>And where we will assign recipient address?

can you paste your rule again?

Hi @reeenz20
I have pasted the full file :
Alert when the rate of events exceeds a threshold
(Optional)
Elasticsearch host
es_host: watchmandev
(Optional)
Elasticsearch port
es_port: 9200
(OptionaL) Connect with SSL to Elasticsearch
#use_ssl: True
(Optional) basic-auth username and password for Elasticsearch
#es_username: anjans
#es_password: Anjan@123
(Required)
Rule name, must be unique
name: Example frequency rule
(Required)
Type of alert.
the frequency rule type alerts when num_events events occur with timeframe time
type: frequency
(Required)
Index to search, wildcard supported
index: logstash-*
(Required, frequency specific)
Alert when this many documents matching the query occur within a timeframe
num_events: 2
(Required, frequency specific)
num_events must occur within this amount of time to trigger an alert
timeframe:
hours: 4
(Required)
A list of Elasticsearch filters used for find events
These filters are joined with AND and nested in a filtered query
For more info: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html
filter:

term:
host.keyword: "ERROR"

(Required)
The alert is use when a match is found
alert:

"email"

(required, email specific)
a list of email addresses to send alerts to
email:
smtp_host: ExternalSMTPRelay
smtp_port: 465
smtp_ssl : true
from_addr<EMAIL_ADDRESS>smtp_auth_file: /opt/elastalert/smtp_auth_file.yml

these lines should be above email:
smtp_host: ExternalSMTPRelay
smtp_port: 465
smtp_ssl : true
from_addr<EMAIL_ADDRESS>smtp_auth_file: /opt/elastalert/smtp_auth_file.yml
should be like this:
alert:

"email"

smtp_host: ExternalSMTPRelay
smtp_port: 465
smtp_ssl : true
from_addr<EMAIL_ADDRESS>smtp_auth_file: /opt/elastalert/smtp_auth_file.yml
email:

<EMAIL_ADDRESS>

Hi @reeenz20
I have tested my SMTP in my Ubuntu Server. It is sending mails successfully. Still I am not getting the elastic search/elastalert alert mails though hits are there from logstash.
What could be the problem. Is there any way to debug why mail is not triggering when I am trying to test the rule. I have included the smtp config as you said above.
The hits output is as follows:
INFO:elastalert:Ignoring match for silenced rule Example frequency rule
Would have written the following documents to writeback index (default is elastalert_status):
silence - {'rule_name': 'Example frequency rule', '@timestamp': datetime.datetime(2017, 5, 22, 17, 33, 12, 395317, tzinfo=tzutc()), 'exponent': 0, 'until': datetime.datetime(2017, 5, 22, 17, 34, 12, 395291, tzinfo=tzutc())}
silence - {'rule_name': 'Example frequency rule', '@timestamp': datetime.datetime(2017, 5, 22, 17, 34, 18, 74940, tzinfo=tzutc()), 'exponent': 0, 'until': datetime.datetime(2017, 5, 22, 17, 35, 18, 74923, tzinfo=tzutc())}
silence - {'rule_name': 'Example frequency rule', '@timestamp': datetime.datetime(2017, 5, 22, 17, 35, 20, 754819, tzinfo=tzutc()), 'exponent': 0, 'until': datetime.datetime(2017, 5, 22, 17, 36, 20, 754802, tzinfo=tzutc())}
silence - {'rule_name': 'Example frequency rule', '@timestamp': datetime.datetime(2017, 5, 22, 17, 36, 23, 189863, tzinfo=tzutc()), 'exponent': 0, 'until': datetime.datetime(2017, 5, 22, 17, 37, 23, 189846, tzinfo=tzutc())}
silence - {'rule_name': 'Example frequency rule', '@timestamp': datetime.datetime(2017, 5, 22, 17, 37, 38, 354880, tzinfo=tzutc()), 'exponent': 0, 'until': datetime.datetime(2017, 5, 22, 17, 38, 38, 354851, tzinfo=tzutc())}
silence - {'rule_name': 'Example frequency rule', '@timestamp': datetime.datetime(2017, 5, 22, 17, 38, 40, 721022, tzinfo=tzutc()), 'exponent': 0, 'until': datetime.datetime(2017, 5, 22, 17, 39, 40, 721008, tzinfo=tzutc())}
elastalert_status - {'hits': 125176, 'matches': 62588, '@timestamp': datetime.datetime(2017, 5, 22, 17, 38, 41, 598797, tzinfo=tzutc()), 'rule_name': 'Example frequency rule', 'starttime': datetime.datetime(2017, 5, 21, 17, 32, 17, 532858, tzinfo=tzutc()), 'endtime': datetime.datetime(2017, 5, 22, 17, 32, 17, 532858, tzinfo=tzutc()), 'time_taken':<PHONE_NUMBER>829773}
Thanks

are you using --verbose or --debug? It wont send messages if your using --debug. Use --verbose instead.

Hi @reeenz20
Where I have to include the --debug or --verbose command.
I was running the test thru :
$ elastalert-test-rule example_rules/example_frequency.yaml
as given in
https://elastalert.readthedocs.io/en/latest/running_elastalert.html#testing-your-rule
Thanks

you might want to read this guide. https://elastalert.readthedocs.io/en/latest/
python -m elastalert.elastalert --verbose --rule example_rules/example_frequency.yaml

elastalert-test-rule won't send alerts, it will just print them to the screen, which would also happen if you added --debug. What @reeenz20 suggested is correct.

Hi @Qmando & @reeenz20
I am getting error now as :
ERROR:root:Error while running alert email: Error connecting to SMTP host: No suitable authentication method found.
Though my SMTP is working fine using telnet in Ubuntu with no authentication required. Also can you explain me what shall be in  smtp_auth_file.yml. Is it .yml file or .yaml file?
By the way , in the example it is mentioned that mail will be triggered. Body of the mail is also written here.
https://elastalert.readthedocs.io/en/latest/running_elastalert.html
Thanks

Check your smtp host. You might wanna try smtp.mail.yahoo.com? Just double check your mail setting.

If it's working fine without authentication, try removing the auth file?
You could also try connecting to non-ssl, though I'm not sure that would make a difference.


Option to return config asynchronously from knexfile.js

I'm porting a suite of applications over to Docker. As part of that process, we're moving all of our configuration into "the cloud." In other words, we're no longer loading our config from flat files that are stored with our application's source code in production... it's all fetched via remote service calls.
I'm now having to consider how I want to handle our application's knexfile.js file. It seems like it would be beneficial if support were added for returning a promise for a config object from this file.

In the meantime, I'm going to look into using something like this.

@tkambler another way around this would be to run a postinstall script which fetches your configs so your application could still rely on local files.

@tkambler I would say that this is not really knex's responsibility to implement. Why do you need to get configuration from knexfile.js ? You can initialize knex like this if you need async config loading:
var knex = null;
resolveConfig.then(function (config) {
  knex = require('knex')(config);
});

If you need also knex client I would say that fetching file to local machine would be good choice... anyways after updating configuration you need to restart application server to update run configuration to your application.

@elhigu What about the use case when knex is used as tool for migrations.
In example knex migrate:latest will use knexfile.js to get configuration. How do I inject it asynchronously?

@ruzpuz In that case, don't use the knex CLI utility. Write your own script using the migration API.

@tkambler sounds a bit disappointing but c'est la vie

If there is lots of demand for this I suppose it could be implemented, but this is so rare use-case that one can use migration API+ own async load routines.

I suppose this would be a 'nice to have' feature. However you are right.


People Finder Improvements: updated filterForSearch and style tweaks

https://github.com/grommet/grommet-hpe/issues/80:
updating config filterForSeach function to account for common names with middle initials (e.g. "Martin R Fink"), by searching for givenName and sn when there's at least 2 words in the search field.
https://github.com/grommet/hpe-design/issues/42:
Updates/validation:

[x] 2. Map location centering that takes into account the location bubble. Offset the v-height to accommodate the blob. Map popup now vertically centered.
[x] 6. Link styling for email and phone should be the same size. Do we need header for email, phone, timezone? Updated timezone to use Paragraph component instead of h3 tag to be more consistent with email and phone spacing.
[x] 7. I thought we got rid of the underline on link styling…Looks odd on the email and phone values. Switching <a> to <Anchor> tags to remove link styling from email and phone numbers. Also removing link underline from close button in leaflet popup.
[x] 8. H1 line-height is shifting text down further from top-margin. Do we need to fix our line height for Metric? H1 is using Heading component, so margin is consistent with the other Paragraphs on Person page.
[x] 9. Do we need to have a message for unavailable phone numbers? Look at Meg Whitman contact. Might be nice to say, "Phone number not available." Switched text from "no phone #" to "Phone number not available" for Person entries with missing phone numbers.
[x] Using person groups index as key to fix same key warning.

perfect you are the best, thank you !


Indexer with type parameter

type MapThing<T> = {
    [x: string]: T; // Property 'length' of type 'number' is not assignable to string index type 'T'.
    length: number;
}
let a: MapThing<number>;
let b: MapThing<string>;

interface MapThingi<T> {
    [x: string]: T; // Property 'length' of type 'number' is not assignable to string index type 'T'.
    length: number;
}
let c: MapThingi<number>;
let d: MapThingi<string>;

A type parameter on an indexer is checked eagerly, should happen on instantiation.

Perhaps, but alternatively, you could type MapThing as:
interface MapThing<T> {
    [x: string]: T | number;
    length: number;
}

or just omit the type parameter if you know you're only going to be using number for T:
interface MapThing {
    [x: string]: number;
    length: number;
}

I would like to think that this:
interface Map<T> {
    [x: string]: T;
}

type MapThing<T> = Map<T> & { length: number; };

Would be the best way to represent that at present. There's no compromises in genericness, signatures, or completion assistance when represented this way.

:+1:
You both got me thinking about:
type MapThingConflict<T> = Map<T> & { length: number; } & { length: string; } & { length: boolean; } ;

Should be an error or a combined/merged type as something like this?
interface MapThingConflict<T> {
    [x: string]: T;
    length: boolean
}

I think the behavior here is effectively calcified


Implement dummy table support

This adress #155
For queries which have select without from clause like select 42. I have implemented selectDummyPlan which just works with the values captured by the select.flds. This way we don't have to refactor all places which relies on select.from since most are interested in select.from.plan which for an unintialized joinRset is nil( see NOTE below) we can then send the selectDummyPlan down the chain.
The previous failed tests are passing but there are new failing tests
$ go test
--- FAIL: TestMemStorage (0.64s)
        storage_test.go:356: 1348 error: internal error 052
--- FAIL: TestFileStorage (2.44s)
        storage_test.go:356: 1348 error: internal error 052
--- FAIL: TestOSFileStorage (1.82s)
        storage_test.go:356: 1348 error: internal error 052
FAIL
exit status 1
FAIL    github.com/cznic/ql     5.071s

I couldn't debug why these tests are failing, so I might need a pointer in the right direction here.
NOTE I have just realised that select.from is  always not nil which makes things a bit complicated as I was expecting it to be nil for the case of missing from clause. That has forced me to use reflection to ensure the select.from object is not initialized.

Thanks for the PR. I will look at the code this evening.

NOTE I have just realised that select.from is always not nil which makes things a bit complicated as I was expecting it to be nil for the case of missing from clause. That has forced me to use reflection to ensure the select.from object is not initialized.

I think using reflect can be avoided by checking for len(s.from.sources) != 0 instead of s.from != nil. But I have not yet seen your code, so this is a blind shot only ;-)

The previous failed tests are passing but there are new failing tests

Test 1348 exercises the new WHERE EXISTS construction, which is yet to be implemented, I guess. But I think it's better to do one thing at a time, so let's just ignore the error for now.

Is there any reason why select.from is always non nil?

I'll now read your code and leave comments, if any, on the individual commits. That might mean commenting on something changed or removed in a later commit. Maybe it would be better for you to work like this:
$ git checkout -b tmp
$ # work on change
$ git commit -a -m 'Commit message'
$ # repeat as necessary
$ git checkout issue155
$ git merge --squash tmp

and then submit a "cumulative" PR. Just an idea, this way is fine as well. In any case, please make the next batch of changes a separate PR, to keep us on an non-overlapping track, thank you.

Is there any reason why select.from is always non nil?

It captures the values of the FROM clause. So far that clause had to be always present in a valid SELECT statement and thus the value was never nil. Some code assumed that invariant, hence your changes ;-)

@gernest Thank you, it's a very good start! However, there's still a long road ahead ;-)

@cznic cool! What do i do now? It looks like I have addressed already all the changes you requested. I will do the squashing thing next time.
The select behavior can stay the same though I don't see  any necessity to change select.from to nil values.

I think you can start thinking about getting testdata.ql tests 1348 and 1349 to work. (The later one seems to be missing a NOT specifier, BTW.) It seems to me like a necessary precursor to "joining the pieces" that will eventually close this issue. Dificulty: intermediate.

Great! At least now I know where to look! I will keep work on this branch. I might push several commits while I'm iterating over stuffs, I will squash them before I ping you for review.

@cznic I have a question, regarding behavior of the test case
maybe can you explain to me the meaning of this query SELECT * FROM t WHERE EXISTS (SELECT * FROM t WHERE i == 2);
and what it will mean when we omit EXISTS or when we replace it with NOT EXISTS ?
I'm still groking SQL so if you give me more context to these will be highly appreciated.

maybe can you explain to me the meaning of this query SELECT * FROM t WHERE EXISTS (SELECT * FROM t WHERE i == 2);

IIUC, this select statement should return all rows from t where the where predicate evaluates to true. exists (select ...) clause should evaluate to true if its parenthesized select statement returns a non zero number of rows.
not exists (select ...) returns true if the select statement returns zero rows.
Both cases should be optimized to bail-out on the first row returned by the parenthesized select, if any.

@cznic I need help understanding this situation.
This  query SELECT * FROM t WHERE EXISTS (SELECT * FROM t WHERE i == 2); is compiled to something like this.
               &ql.selectStmt{
                    distinct: false,
                    flds:     {
                    },
                    from: &ql.joinRset{
                        sources: {
                            []interface {}{
                                "t",
                                "",
                            },
                        },
                        typ: 0,
                        on:  nil,
                    },
                    group:         (*ql.groupByRset)(nil),
                    hasAggregates: false,
                    limit:         (*ql.limitRset)(nil),
                    offset:        (*ql.offsetRset)(nil),
                    order:         (*ql.orderByRset)(nil),
                    where:         &ql.whereRset{
                        expr: nil,
                        src:  nil,
                        sel:  &ql.selectStmt{
                            distinct: false,
                            flds:     {
                            },
                            from: &ql.joinRset{
                                sources: {
                                    []interface {}{
                                        "t",
                                        "",
                                    },
                                },
                                typ: 0,
                                on:  nil,
                            },
                            group:         (*ql.groupByRset)(nil),
                            hasAggregates: false,
                            limit:         (*ql.limitRset)(nil),
                            offset:        (*ql.offsetRset)(nil),
                            order:         (*ql.orderByRset)(nil),
                            where:         &ql.whereRset{
                                expr: &ql.binaryOperation{
                                    op: 57378,
                                    l:  &ql.ident{s:"i"},
                                    r:  ql.value{
                                        val: ql.idealInt(2),
                                    },
                                },
                                src:    nil,
                                sel:    (*ql.selectStmt)(nil),
                                exists: false,
                            },
                        },
                        exists: true,
                    }

What I see here is, on select.where the expr is nil . I feel like this is okay, since we now rely on select.where.sel to be acting like the expr field with respect to the nature on the query.
But on the test case at https://github.com/cznic/ql/blob/master/all_test.go#L2805 we are insisting only  passing expr which in this situation is nil resulting into the test case to falsely fail. My current state of mind thinks the test case need to be adjusted to accommodate the new features. I have run out of ideas regarding this. And obviously I'm supposed to make the test pass not refactor the tests not to fail.
I added the implementation taking care of the where.exists semantics as you suggested but that part of the code is never reached( In ine whereRset.plan)

But on the test case at https://github.com/cznic/ql/blob/master/all_test.go#L2805 we are insisting only passing expr which in this situation is nil resulting into the test case to falsely fail.

I think your observation is correct. This
if w := x.where; w != nil {
	mentionedColumns(w.expr)
}

should be replaced with something like
if w := x.where; w != nil {
	if e := w.expr; e != nil {
		mentionedColumns(w.expr)
	}
	if s := w.sel; s != nil {
		if err := testMentionedColumns(s); err != nil {
			return err
		}
	}
}

And obviously I'm supposed to make the test pass not refactor the tests not to fail.

Yes please ;-) But it applies to the point where we will finally merge branch issue155 to master. While working on the branch, it's okay to break tests because some old invariants hold no more. The go test command will kindly remind us to fix all those places before closing #155.

I added the implementation taking care of the where.exists semantics as you suggested but that part of the code is never reached( In ine whereRset.plan)

Don't worry, we will figure it out. But let's look into this after we close the current "round". That is: Please address the few review comments above so we can "bump" this issue to the next level (and the next PR) by merging the pending PR into branch issue155. Once again: It does not have to work/work completely yet. The point is in making progress towards that :+1:
Thank you for your work!

@cznic It looks like I got lucky and made where exists to work.
I tried to squash and i messed up almost thought i lost my changes(panicked), hope you don't mind the hassle until i master the git wizardly.

Thanks for the updates, I hope to review them this evening.

It looks like I got lucky and made where exists to work.

Congrats!
Note: We should not continue with this PR as "rolling update". After we will agree this PR is okay in its state as of now (or after some adjustements), we shall merge it into branch issue155 and continue with any remaining steps in another PR, zero or more of them as necessary, all of them will be separately meerged into branch issue155.
It's good to keep PRs bite sized and this is rather a big change. You may notice the Go project also does bigger changes in series of CLs. Think of a PR the same as a CL. Sometimes one is enough, but there's never a need to squeeze too much into a single one.
Later, the final merge of branch issue155 into branch master can be --squashed, if you prefer, to make it a single change in the history, if that makes you worry. Worry not ;-)

I tried to squash and i messed up almost thought i lost my changes(panicked), hope you don't mind the hassle until i master the git wizardly.

No problems caused.

@gernest Thank you for great work! Please take a look at the latest review comments. Once resolved, I believe this PR can be merged to the issue155 branch.
The next step, in the next PR, is to extend testdata.ql with more tests. For example, there's no test yet using WHERE NOT EXISTS. I will probably write them, but please feel free to come with your own. More tests with more and different ideas are good.
I am impressed how fast you got familiar with ql internals. Very well done! :+1:

@cznic thanks for the compliment!
everything on my branch looks great at the moment, I hope this will be merged sooon. I find the ql code base to be a pleasure to navigate, only the parser generation part looks alien to me ( the grammar and .y files)
I will wait for your direction on the next phase for closing  #155 . I can't come up with best test cases at the moment as I still lack the big picture and haven't touched most of the codebase yet.

everything on my branch looks great at the moment, I hope this will be merged sooon.

Probably today late evening.


Gh 1243 operation chain handler

Codecov Report

Merging #1453 into develop will increase coverage by 0.26%.
The diff coverage is 78.1%.

@@             Coverage Diff              @@
##             develop   #1453      +/-   ##
============================================
+ Coverage      66.14%   66.4%   +0.26%
- Complexity      1646    1683      +37
============================================
  Files            713     713
  Lines          23214   23289      +75
  Branches        2940    2963      +23
============================================
+ Hits           15354   15465     +111
+ Misses          6668    6633      -35
+ Partials        1192    1191       -1

Impacted Files
Coverage Δ
Complexity Δ

...fer/data/elementdefinition/ElementDefinitions.java
59.55% <ø> (ø)
0 <0> (ø)
:arrow_down:

...gchq/gaffer/federatedstore/operation/AddGraph.java
93.54% <ø> (ø)
0 <0> (ø)
:arrow_down:

...q/gaffer/federatedstore/operation/RemoveGraph.java
100% <ø> (ø)
0 <0> (ø)
:arrow_down:

...affer/federatedstore/operation/GetAllGraphIds.java
100% <ø> (+8.33%)
0 <0> (ø)
:arrow_down:

...store/operation/handler/OperationChainHandler.java
65.71% <ø> (ø)
0 <0> (ø)
:arrow_down:

...n/java/uk/gov/gchq/gaffer/operation/Operation.java
59.57% <0%> (+2.43%)
9 <0> (ø)
:arrow_down:

...r/federatedstore/operation/FederatedOperation.java
100% <100%> (ø)
0 <0> (?)

...e/operation/handler/FederatedOperationHandler.java
91.66% <100%> (ø)
0 <0> (ø)
:arrow_down:

...fer/operation/serialisation/TypeReferenceImpl.java
65.62% <100%> (+65.62%)
1 <0> (+1)
:arrow_up:

...ation/handler/FederatedOperationOutputHandler.java
100% <100%> (ø)
0 <0> (ø)
:arrow_down:

... and 44 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5826be8...1a13069. Read the comment docs.

Just need to make a couple of changes.


matplotlib.cm.rainbow cannot be resolved

Environment data

Language Server version: vscode python extension v2020.6.89148
OS and version: windows 10
Python version (& distribution if applicable, e.g. Anaconda): 3.7.7

Expected behaviour
create an conda env with python 3.7
pip install matplotlib==3.2.2
then create py file will flowing code.
import matplotlib.cm as cm
cm.rainbow(xxxx)
it says Module 'matplotlib.cm' has no 'rainbow' member
Actual behaviour
XXX
Logs

XXX

Code Snippet / Additional lnformation

XXX

to clarify auto complete, and go to definition also not working.
but run time passes.


Fulltext search docs improvement

Currently we have the following docs for full text search/indexes:

Indexes concepts page has a section on Full-text indexes for MySQL and MongoDB
Full-text search concepts page for PostgreSQL and MySQL
Client API reference entry forsearch for PostgreSQL and MySQL
Filtering and sorting concepts page has a Sort by relevance section for PostgreSQL

Missing:

documentation of @fulltext in the schema reference attributes section
document index limitation for MongoDB - see prisma/prisma#2729
MongoDB full text search?
PostgreSQL full text indexes?
links to index configuration docs from client full text search docs - see comment here

Current state:

MongoDB full text search?
Not implemented yet.

PostgreSQL full text indexes?
Not adequately implemented yet to be worth documenting.

Note that _relevance is related to fullTextSearch and has a docs issue as well: https://github.com/prisma/prisma/issues/12249

👋 we're cleaning up our backlog of issues and are closing everything older than two years. If you still feel that this issue is relevant, please feel free to re-open and our team will take a look. Thank you!


Use git commit hash along with version number

Include the commit hash alongside the version number.
Commit hash is valuable to developers for tracing the currently executing code to a specific comment during development/debugging.
Version numbers are useful for users to understand if they're using a new/old version of the software.
New version number example:

v1.0.1 ccd29824e5fdb25dcb82931293jdsdi23da90ja

fixed 48c858e8178749ba09b02285be7aede26e860e9e


Don't emit stack traces when command-line / config file parsing fails

I used the ConfigException workaround because I can't do this:
    Config config;
    try
    {
        config = parseConfigFile(cmdln); /// <= Error: reassigning const/immutable fields
        if (cmdln.config_check)
        {
            writefln("Config file '%s' succesfully parsed.", cmdln.config_path);
            return 0;
        }
    }
    catch (Exception ex)
    {
        writefln("Failed to parse config file '%s'. Error: %s",
            cmdln.config_path, ex.msg);
        return 1;
    }

    auto node = new Node(config);
    return runEventLoop();
}

Rebased.

@Geod24 rebased and green.

Fixed the two issues.

Rebase?

There's another issue I've uncovered here. I cannot pass ex.message to the constructor of ConfigException because it expects a string, not a const(char)[]. Sigh.

Let's just go with msg then and reassess if something goes wrong. Worst case scenario we got an empty message...

Yeah, maybe this can be fixed upstream though. Anyway rebased!

Ready for merge.


Fix C# operator *(Transform3D, Aabb)

The current C# implemention of static Aabb operator *(Transform3D transform, Aabb aabb) (added in #64729) is pretty-much a copy-paste of the core/C++ implementation of AABB Transform3D::xform(const AABB &p_aabb). The issue is that such implemenation uses Basis[int] operator and:

in core/C++ this operator returns Basis row,
in GDscript/C# this operator returns Basis column (axis).

This discrepancy was not taken into account, hence incorrect calculations. This PR fixes that.
Fixes #97197.

Thanks!


Fix the getNext method in the TelegrafExecutionContext class

Hey @Maks1mS, good bugfix! Can you rebase you branch from main? We have some problems after repo relocation)

Oops! I will create a new one.

The new PR is #909.


400ms bt Issue

Yo, I added atm 400ms backtrack from pull request.
I diagnostic the code and its crashing cause:
File: e:\visual studio\vc\tools\msvc\14.16.27023\include\deque
Line: 315
Expression: cannot dereference value-initialized deque iterator
Problem is with: record && record->size() && Backtrack::valid(record->front().simulationTime)
Maybe cause getLerp() // An invalid parameter was passed to a function that considers invalid parameters fatal.

I fixed it.


add unsupported op Equal and ExpandDims in lite

Can one of the admins verify this patch?

I signed it!


Do not see startsample.sh after build

IMPORTANT: Before you create an issue, please take a look at our Issue Reporting Guide.
Briefly summarize your issue:

After finishing bash ./setup.sh config.txt
I did not see startsample.sh in the directory neither running bash startsample.sh returns
"bash: startsample.sh: No such file or directory"
What is the expected behavior?

What behavior are you observing?

Provide the steps to reproduce the issue, if applicable:

Tell us about your environment:

What version of the AVS Device SDK are you using?
  <x.y.z>

Tell us what hardware you're using:

[ ] Desktop / Laptop
[ ] Raspberry Pi
[ x ] Other - tell us more:

Tell us about your OS (Type & version):

[ ] Linux
[ X ] MacOS
[ ] Raspbian Stretch
[ ] Raspbian Jessy
[ ] Other - tell us more: Android

@ruradium , which guide are you following for the steps? What steps have your performed?

i am following the wiki guide for raspberry pi and cannot fine the startsample.sh after running setup.sh


[sw, dif pwrmgr smoketest] Deadlock when disabling USB clock in active power

In dif_pwrmgr_smoketest.c, the following line disables everything, including USB clock in active power, before transitioning to low power mode:
https://github.com/lowRISC/opentitan/blob/efb811bdb2bca2e2e6113f84cb9ca0bd0b4fd8f3/sw/device/tests/dif/dif_pwrmgr_smoketest.c#L95
Here's the waveform snippet:

This results in a deadlock when the pwrmgr triggers a WFI.
pwrmgr transitions to FastPwrStateDisClks and waits for the clkmgr to disable all clocks. The latter is unable to disable the USB clock because it is already off. The prim_clock_gating_sync in the clkmgr is no longer operational because clk_usb_i is already off:
https://github.com/lowRISC/opentitan/blob/efb811bdb2bca2e2e6113f84cb9ca0bd0b4fd8f3/hw/top_earlgrey/ip/clkmgr/rtl/autogen/clkmgr.sv#L319
Consequently, the clkmgr keeps its clk_status asserted and pwrmgr never transitions out of FastPwrStateDisClks state.

Key things to note here:

This test should be have worked on Verilator - if it did, then there is a hole somewhere that needs to be plugged!
We should document in pwrmgr spec that the USB clock in active power must not be turned off

The correct fix for this is probably in the clkmgr.

Updated the SW test to not disable USB clock in active power to get around this bug.
https://github.com/lowRISC/opentitan/pull/6500/commits/ea4789518e646e665d3fdcd4f8a4d82727160346

let me take a look at this one.
The usb disable as you say should probably be exposed to clkmgr and short circuit around the check.

let me take a look at this one.
The usb disable as you say should probably be exposed to clkmgr and short circuit around the check.

Welcome back!

I've thought a bit about this, and the right way to address it, i believe, is to have a handshake relationship between pwrmgr / clkmgr on each of these root clocks.
Is that an acceptable change to DV right now? Basically what would happen is that the pwrmgr ip_clk_en that goes to clkmgr would split into one for each root clock source (main, io, usb).  So if someone were to turn off the usb before low power entry, the current handshake routine would be initiated earlier.
Let me know what you guys think.  If that's too much change, we can also make a temporary fix, and table the real fix to a later date / enhancement.

So you are saying a dedicated per clk ip_clk_en would trigger changes in clk_status. So if there is no change in a given clock then its contribution to clk_status doesn't need to change. Something like this would work, and the question is when we can get it fixed :)


Support enableIngress for RayCluster

address #27
Add new reconcile loop for ingress object. This is only working for RayCluster with spec.HeadGroupSpec.EnabledIngress = true users. By default, this is not enabled.

LGTM, thanks!


Remove reliance on user's account balance

When Zonky moves to implement PSD2, we will lose access to all financial information, incl. user's account balance. This change removes all reliance on it from the robot.

Sonar failed due to some weird JVM SIGSEGV. Can reproduce locally. Will have to be investigated independently.


Marking as minimum upstream severity instead of max

Description
Luigi marks a wrapper task as UPSTREAM_FAILED or UPSTREAM_DISABLED when ANY of its upstream task is FAILED or DISABLED.  This should be ALL instead of ANY
Motivation and Context
Luigi marks a wrapper task as UPSTREAM_FAILED or UPSTREAM_DISABLED when ANY of its upstream task is FAILED or DISABLED. Moreover, when the wrapper task is marked as UPSTREAM_DISABLED, luigi shut the worker down as it should do. This causes another PENDING_TASK in the wrapper task not to run because worker is down. This behavior should change. Marking as UPSTREAM_ should be ALL instead of ANY.
It is discussed in PR #1782 . For more detail about the problem, please check it.
This is also related #871.
Have you tested this? If so, how?
Using tox locally and debugging it in PyCharm.

Okay. Let me try to write that test case now and verify that the behaviour changes as I expected it. If it does I merge this followed by sending a PR with my own test case.

There's really only one narrow case where this PR keeps workers alive: if all runnable tasks are currently being run by other workers. If all of the tasks finish (success or fail) then the worker will die. This will only make a difference if you rely on retrying often and expect new tasks to switch from FAILED to PENDING while the tasks are running.
It seems that the identical effect could be achieved by simply adding a configuration parameter to count RUNNING tasks in the PENDING count in order to keep workers alive. Alternatively, we could add a timeout where workers stay alive until a configurable amount of time has passed without seeing any running tasks.
Meanwhile, my visualizer now gives a misleading under-count of the number of upstream disabled and failed jobs. It seems quite weird that this count should grow as more jobs complete successfully. If an important job is being blocked by a failure, I don't want to have to wait until every other dependency is complete to learn about it.
Am I missing something here?
Also, note that this change is equivalent to just changing the UPSTREAM_SEVERITY_ORDER to
UPSTREAM_SEVERITY_ORDER = (
    '',
    UPSTREAM_DISABLED,
    UPSTREAM_FAILED,
    UPSTREAM_MISSING_INPUT,
    UPSTREAM_RUNNING,
)

Which I think is pretty weird, as it makes depending on a RUNNING job more severe than depending on a DISABLED job.

Meanwhile, my visualizer now gives a misleading under-count of the number of upstream disabled and failed jobs. It seems quite weird that this count should grow as more jobs complete successfully. If an important job is being blocked by a failure, I don't want to have to wait until every other dependency is complete to learn about it.

Am I missing something here?

What you say is correct. The interpretation of "task X is an upstream disabled task" changed with this patch:

Before it meant "task X has some dependency which is disabled* (but no dependency is worse off)"
Now it reads "Nothing on X or it's dependencies is progressing! All non-complete dependencies of X are (at least) disabled*".

The *star because it's actually disabled+upstream disabled.

I think both numbers are pretty definitions are hard to grasp and many luigi users don't care to learn the difference. But if any, I actually do agree that the former is more intuitive. I'm not sure if it's better though. The old definition is very sensitive. Imagine a big enterprise and a leaf task from team Z has failed, now team A knows nothing about the details of team Z's tasks, but they transitively depend on it. But still their task is marked as reddish even though most dependencies are progressing. However, team Z is probably fully aware of the problem, they've got their error emails and can see that their task is "FAILED" in the scheduler.
I'm kind of open to reverting this if you really think the old way to count is more useful. As long as the test case from #1790 still passes.
The only thing I'm against is making the two ways of counting configurable, as it is too detailed for the typical luigi user to care about.

My point of view is that if any upstream task is failed or disabled, your
task will not run. It's a bit weird to not mark it reddish until it
completely stops getting new dependencies run. Seeing an accurate upstream
failed count makes it easier to know that there's a serious problem. Seeing
a job you're tracking marked as upstream failed tells you that there's
something you need to look into to make sure it gets done. Maybe you need
to fix it, maybe you need to ping another team to make sure they're on it.
I'll put something together to handle the test case. I didn't see that was
there!
On Thu, Nov 10, 2016 at 6:57 PM, Arash Rouhani<EMAIL_ADDRESS>wrote:

Meanwhile, my visualizer now gives a misleading under-count of the number
of upstream disabled and failed jobs. It seems quite weird that this count
should grow as more jobs complete successfully. If an important job is
being blocked by a failure, I don't want to have to wait until every other
dependency is complete to learn about it.
Am I missing something here?
What you say is correct. The interpretation of "task X is an upstream
disabled task" changed with this patch:

Before it meant "task X has some dependency which is disabled* (but
no dependency is worse off)"
Now it reads "Nothing on X or it's dependencies is progressing! All
non-complete dependencies of X are (at least) disabled*".

The *star because it's actually disabled+upstream disabled.
I think both numbers are pretty definitions are hard to grasp and many
luigi users don't care to learn the difference. But if any, I actually do
agree that the former is more intuitive. I'm not sure if it's better
though. The old definition is very sensitive. Imagine a big enterprise and
a leaf task from team Z has failed, now team A knows nothing about the
details of team Z's tasks, but they transitively depend on it. But still
their task is marked as reddish even though most dependencies are
progressing. However, team Z is probably fully aware of the problem,
they've got their error emails and can see that their task is "FAILED" in
the scheduler.
I'm kind of open to reverting this if you really think the old way to
count is more useful. As long as the test case from #1790
https://github.com/spotify/luigi/pull/1790 still passes.
The only thing I'm against is making the two ways of counting
configurable, as it is too detailed for the typical luigi user to care
about.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/spotify/luigi/pull/1789#issuecomment-259869001, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AB_rbemSxz1GM5dKsKmJ2lGWSYILVKPuks5q89mngaJpZM4JW6Rw
.

Seeing an accurate upstream
failed count makes it easier to know that there's a serious problem. Seeing
a job you're tracking marked as upstream failed tells you that there's
something you need to look into to make sure it gets done

@daveFNbuck  you are right, but upstreams are not just used for that; one of upstream status which is UPSTREAM_DISABLED, is also used to decide whether workers should keep working or not. I mean, this is not just used to mark it as "hey there is a problem, lets take a look". (There is no problem with UPSTREAM_FAILED by the way.). Whenever a wrapper task is UPSTREAM_DISABLED, this also means making luigi stops working. This is why we change the old behavior which was calculating upstream status as MAX among upstreams to MIN.
            if task.status == PENDING and in_workers:
                upstream_status = self._upstream_status(task.id, upstream_table)
                if upstream_status != UPSTREAM_DISABLED:
                    locally_pending_tasks += 1
                    if len(task.workers) == 1 and not assistant:
                        n_unique_pending += 1

I understand what you mean, but isn't not letting workers keep working for a task needs to be retried when there is no pending task, is problematic? I think, this is more critical issue that not to let it to retry a task in this case.
If you guys think that kind of monitoring that you mentioned is also important which is I also agree, I think we should do it in a way by not reverting the feature in this PR.

Thanks for everyone's input. My summary so far would be that we want these features in luigi:

The visualizer should indicate if there's anything at all downstream for a task that's DISABLED, as that thing most probably requires human intervention to get resolved.
keep-alive-Workers should be smart and not quit just because there's failed stuff left, but they should give up on DISABLED.

An idea for solution could be to

Remove UPSTREAM_FAILURE from the visualizer. People shouldn't care about it. Failures are normal and people (for dependent tasks) shouldn't be alarmed unnecessarily.
Introduce something called BLOCKED_DISABLED. UPSTREAM_DISABLED is the more sensitive ANY-version and BLOCKED_DISABLED is the ALL-version.
Perhaps BLOCKED_DISABLED doesn't need to be in the visualizer as it might be confusing. I'm not sure about this.

I have a different solution that I think fixes the issues. We just count FAILED tasks as pending in count_pending in the scheduler. That way we keep the worker alive until all jobs are fully blocked by disabled jobs. I have a branch that does this, but it breaks some worker tests because they assume everything will stop once all jobs are failed. So it'll take a bit of effort to fix each of those appropriately. With this, we don't need BLOCKED_DISABLED, as we're more directly getting at the idea of allowing re-runs until jobs are disabled.
I personally like having UPSTREAM_FAILURE but I can see your side too. Someone on my team is also currently adding a READY state to our visualizer that will filter to which just shows PENDING tasks where every dependency is DONE. I think this is more useful than UPSTREAM_FAILURE and keeps the nice balance of 8 items in two rows of 4 if we remove a status when we add this.

@daveFNbuck:
As for count-FAILED-as-pending idea. It clings well in my ears, probably because we make clear the intention that they will be re-run.
As for the READY state, I also like it, BLOCKED_DISABLED is a bit long. As you pointed out. We should always keep the 8 items balance in mind too. :)
@daveFNbuck, were you up for eventually implementing any of these ideas?

Yeah, I have an implementation mostly done. I'll be on vacation this week,
but I'll pick it up again after that.
On Sun, Nov 13, 2016 at 6:36 PM, Arash Rouhani<EMAIL_ADDRESS>wrote:

@daveFNbuck https://github.com/daveFNbuck:
As for count-FAILED-as-pending idea. It clings well in my ears, probably
because we make clear the intention that they will be re-run.
As for the READY state, I also like it, BLOCKED_DISABLED is a bit long.
As you pointed out. We should always keep the 8 items balance in mind too.
:)
@daveFNbuck https://github.com/daveFNbuck, were you up for eventually
implementing any of these ideas?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/spotify/luigi/pull/1789#issuecomment-260235632, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AB_rbY6TK4_f682ucCM-obMuiYsD4ocRks5q98kmgaJpZM4JW6Rw
.


Issue 100 ft.info

I have updated the library to Jedis 3.4.1 and use the new encode method
Note: I was not able to run all the tests using mvn test (even on master branch) but all test are passing from the ide

@gkorland can please review

@gkorland can please review


Sort facetsDistribution

How can I sort facetsDistribution object and facets array? Expected it to be sorted by descending count values. But I get random order every request. I tested with meilisearch-rails gem.

Hello @molfar!
You cannot do it from the core-engine settings at the moment: you have to do it on your side.


CFFL

CFFL

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_9c3y01vyu8/clone/recipe/meta.yaml:15: pip_install_args: pip install should be run with --no-deps.
ERROR: /tmp/abs_9c3y01vyu8/clone/recipe/meta.yaml:20: missing_wheel: For pypi packages, wheel should be present in the host section
WARNING: /tmp/abs_9c3y01vyu8/clone/recipe/meta.yaml:42: missing_description: The recipe is missing a description
ERROR: /tmp/abs_9c3y01vyu8/clone/recipe/meta.yaml:42: missing_documentation: The recipe is missing a doc_url or doc_source_url
Errors were found


A date divider like in Windows Explorer's Downloads folder

What's the Problem?
I'd like to see the date divider implemented in the Files app. What i mean by date divider is the divider in the Downloads folder in Windows Explorer like in the attachment

Solution/Idea
A toggleable setting to enable or disable the divider, with or without a date
Alternatives
anything similar to a divider, just to make it more pleasing to the eye.
Priorities

Capability
Priority

This proposal will allow developers to see a clear line in between files
Must

This proposal will allow developers to see a clear line in between files indicating the date when the files were made or modified
Could

Files Version
No response
Windows Version
No response
Comments
No response

There is

There is in the menu from the two arrows. just selected grouping by data created or modified

Thank you so much! I couldn't find it in the settings panel so i thought it didn't have this feature but i guess i was wrong


[common][server] Bug fixes and clean of ACL handlers for GRPC

Summary

Consolidate boiler plate code in ACL handlers for access validations
Fix code path that throws exception in GRPC and close the calls with the appropriate error messages
Fix the flow that invokes onMessage twice in GRPC call intercept to prevent exceptions
Rename GrpcSSLUtils to GrpcUtils to accommodate general utility methods
Add reflection service to GRPC server to interact with Grpcurl
Reuse the header for skipping ACLs from router since the store ACL handling is performed at the router for non FC clients
[code readability] Move public methods together followed by overridable and protected methods followed by package private and private methods

How was this PR tested?

Unit tests
Integration tests
Deployment to certification cluster to validate the flow using sample venice application

Does this PR introduce any user-facing changes?

[X] No. You can skip the rest of this section.
[ ] Yes. Make sure to explain your proposed changes and call out the behavior change.

@xunyin8 I think i narrowed down the issue with mockito-inline upgrade which doesn't play well with the bytebuddy we pull in transitively. For now, reverting the mockito dependency upgrade and removed the test that requires mocking static utility.
Will have a separate PR to fix the mocking issue (for mocking static and final classes)


Error Ticker When Ticker is Show and We Click Back

hello,
i found a bug (maybe), when we click the dropdown, while the dropdown showing data, then click back to the previouse page, and suddenly its triggred an error..
FlutterError (_DropDownTextFieldState#6aa21(tickers: tracking 1 ticker) was disposed with an active Ticker.

hope you can help me..

I'm getting the same error too.  I notice that the AnimationController isn't disposed in the dispose() function of the _DropDownTextFieldState class.  However disposing of this still has the same bug - I'm not quite sure how to remove the ticker before disposing.  Anyone?
@override
  void dispose() {
    if (widget.searchFocusNode == null) _searchFocusNode.dispose();
    if (widget.textFieldFocusNode == null) _textFieldFocusNode.dispose();
    _controller.dispose();  // <--- this added here
    _cnt.dispose();
    super.dispose();
  }

Thank you for the fix, @dankhan-helmet. I have included it in the latest release. Sorry for delayed response, @curlylazy . I was bus with other things and couldn't maintain this package. The issue has been fixed in the latest release, dropdown_textfield: ^1.1.0

thank you so much for fixing the package.

Awesome thanks so much for the fix.  Just to note that this update only supports flutter >= v3.22 only, due to MaterialStateProperty -> WidgetStateProperty breaking changes.


TS-1012: Remove BBAN property

Platform team has removed BBAN property from the payment initiation request and no longer accepts it

Don't Czech banks use this property?


Canonicalize URLs for prefix www.

And resort the URL entries for

// Keep this list sorted for easy checking.

@rsc gentle ping :) would appreciate if you can take a look at this


excep handling and F usage update pull

Updates/improvements<PHONE_NUMBER>:

function F development, inclusion and usage within functions cbrt, to_depressed and cbdelta, mainly consisting on setting values to fractions.Fraction object and more fractions module operations.
Inclusion of calling of functions check_types and check_vals for checking on the supportability of the input variables to be passed (variable type or variable matching values). These two functions are soon to be included on exceptions.py.

Also, the commit included the import of fractions module and removing numpy importing as it is not used yet.


xmile files are not uploading, "failed to load external entity"

Whenever I try to upload the xmile file, I get this issue OSError: Error reading file 'Vidzeme.xmile': failed to load external entity "Vidzeme.xmile"
I'm trying to convert the Stella model to Python in order to generate GeoJSON files and visualize the model results in GIS, unfortunately, I don't know much about Stella itself and didn't make the model in question, so I initially assumed, that the problem is in the model since it is a quite complex one, so I checked it with some basic models, and it's not loading either.
I installed PySD via pip. There are currently some global issues going on with Numpy, when I tried to run the code in PyCharm, I got this issue, but it doesn't seem to be a problem, running the same code in VS Code.
I tried to go through the library files separately and most of them give attempted relative import beyond top-level package pylint error even if there is no obvious problem with the package hierarchy.
TIA!

HI @marika-kac
We have rewritten the xmile translator. Do you want to try again your model?