handling exception in getSelection

Description
Rangy can throw in some cases an exception on the getSelection because the nativeSelection is not available.
    at refreshSelection (VM42172 rangy-core.js:3428)
    at WrappedSelection.selProto.refresh (VM42172 rangy-core.js:3462)
    at Object.getSelection (VM42172 rangy-core.js:3223)
    at SelectionWatcher.syncSelection (VM42222 selection-watcher.js:83)
    at SelectionWatcher.getRangeContainer (VM42222 selection-watcher.js:93)
    at SelectionWatcher.getFreshSelection (VM42222 selection-watcher.js:127)
    at Editable.getSelection (VM42151 core.js:365)
    at Highlighting.safeHighlightMatches (VM42272 highlighting.js:217)
    at eval (VM42272 highlighting.js:207)
    at eval (VM42273 spellcheck-service.js:64) "Possibly unhandled rejection: {}"

Changelog

catch rangy exception and return empty RangeContainer

:tada: This PR is included in version 2.1.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


EvalSymlinks: too many links

I'm getting an EvalSymlinks: too many links error when trying to generate a client:
CODEGEN_PKG=../../../k8s.io/code-generator hack/update-codegen.sh

on this repo.
While looking into this, it seems that the error happens when generating code for the fake package:
❯ env CODEGEN_PKG=../../../k8s.io/code-generator hack/update-codegen.sh
Generating deepcopy funcs
path /home/asymmetric/code/go/src/github.com/habitat-sh/habitat-operator/pkg/apis/habitat/v1beta1
Generating clientset for habitat:v1beta1 at github.com/habitat-sh/habitat-operator/pkg/client/clientset
path /home/asymmetric/code/go/src/github.com/habitat-sh/habitat-operator/pkg/client/clientset/versioned
path /home/asymmetric/code/go/src/github.com/habitat-sh/habitat-operator/pkg/client/clientset/versioned/scheme
path /home/asymmetric/code/go/src/github.com/habitat-sh/habitat-operator/pkg/client/clientset/versioned/fake
EvalSymlinks: too many links

(Note: I added the debugging statement that prints path).
This seems to be happening here, which is called indirectly as part of the client code generation.
❯ uname -omr
4.13.0-37-generic x86_64 GNU/Linux

Not sure if this helps, let me know if you need additional info or if I should open the issue on gengo or kubernetes.

+1

I realized that this was happening because I was running the master branch generator against a client-go v6.0.0 codebase.
As soon as I switched to release-1.9, the error went away.

@asymmetric thanks, this saved me some time! :)
@nikhita @sttts Could we maybe add the above to the README? Would be super helpful IMO.

@asymmetric I'm not able to reproduce this on the repo you mentioned: https://github.com/kinvolk/habitat-operator/tree/14ae480d50bae24dd3260a0ade8d0300da3f387b.
@asymmetric @LiliC Is it reproduceable for you always? (seems like it should be though)
And this error occurs if the number of symlinks to evaluate is >255:
https://golang.org/src/path/filepath/symlink.go#L37

Could we maybe add the above to the README? Would be super helpful IMO.

What exactly?

+1 @asymmetric I have the following dependencies and I am still getting this error.
k8s.io/code-generator branch release-1.9
k8s.io/client-go v6.0.0

An older version of golang.org/x/tools/imports (used in the code generator) not only produces this error upon encountering circular symlinks, but also scans packages in your GOPATH which it has no business scanning.
I encountered the issue on golang.org/x/tools/imports revision ce871d178848e3eea1e8795e5cfb74053dde4bb9. The github.com/snapcore/snapd repo in my GOPATH conveniently contains a hell directory which is mean to test for symlink handling bugs.
Updating my local golang.org/x/tools/imports solved this issue for me.


USER 1001 -> USER 1000

A strange docker directive of

USER 1001

when no such user exists in the /etc/passwd of the containers seems to break pyspark. Setting that to USER 1000 (jovyan, the default user) seems to at least let pyspark load it's context.
A side effect of this is that the user sees the username "i-have-no-name" and whoami says it doesn't know user number 1001.
After the dust settles and this has been tested to work, we could consider moving all of them to USER 1000.

I have a feeling that because OS sets a random UID for users inside containers the UID is set to a non-existing 1001 to simulate that effect.


MicronautRequestHandler no longer working with SQS and Jackson

Expected Behavior
After bumping to Micronaut 4 we've experienced some weird stuff regarding serialization and deserialization. For reasons, we cannot use Serde in our project (that has been live in production for more than two years). We want to keep depending on Jackson as before.
I've created a small POC, see "Example Application" below. The basis of the project were created using Micronaut Launch, see micronaut-cli.yml for launch details.
Actual Behaviour
When consuming SQS Events on a Lambda, it fails with a NullPointerException due to the incoming SQSEvent (or intenral content) being null.
Steps To Reproduce

See https://github.com/thobiaskarlsson/micronaut-4-jackson-sqs-issue
Deploy this JAR to AWS
Connect the com.example.SqsHandler as an event handler to an SQS queue as event source.
Fire any SQS message on the queue
Lambda function fails, with the following exception:

Cannot invoke "java.util.List.iterator()" because the return value of "com.amazonaws.services.lambda.runtime.events.SQSEvent.getRecords()" is null: java.lang.NullPointerException
java.lang.NullPointerException: Cannot invoke "java.util.List.iterator()" because the return value of "com.amazonaws.services.lambda.runtime.events.SQSEvent.getRecords()" is null
	at com.example.SqsHandler.execute(SqsHandler.java:29)
	at com.example.SqsHandler.execute(SqsHandler.java:15)
	at io.micronaut.function.aws.MicronautRequestHandler.handleRequest(MicronautRequestHandler.java:110)

Please note that we've managed to reproduce where the SQSEvent (main request object) was null, but it seems to have switched to the getRecords() being null now instead. Either way, it's very strange imo.
Environment Information

JDK 17

Example Application
https://github.com/thobiaskarlsson/micronaut-4-jackson-sqs-issue
Version
4.1.4

Is there any way to write a Unit Test based on raw JSON input/output instead of manually building for instance SQSEvent and invoking the handler function that takes the POJO as input/output? Example:
  // Given
  String rawJsonInput = readJsonFile(...);

  // When
  String rawJsonOutput = handler.execute(rawJsonInput);

  // Then
  JsonApprovals.verifyJson(rawJsonOutput);

If this was possible for all Request Handlers then we could verify this bug locally instead of in runtime. Any thoughts?

@sdelamo, I've tracked it a bit further, and it looks like the ObjectMapper does not deserialize the Records (title-cased) correctly into the records field in SQSEvent, hence get getRecords() returns null. However, in the application.yml we've enabled the acceptCaseInsensitiveProperties but it doesn't seem to "hook" no more. Has there been any changes to how we can configure ObjectMapper from the application config? This works in Micronaut version <4.
jackson:
  mapper:
    acceptCaseInsensitiveProperties: true


Cases for Jetbrains Rider's with Unity3d

Reasons for making this change:
Cases for Jetbrains Rider's user specific stuff
For people using JetBrains Rider as a replacement for Visual Studio in Unity3D
Links to documentation supporting these rule changes:
https://github.com/github/gitignore/blob/master/Global/JetBrains.gitignore

Thanks, but see Global\JetBrains.gitignore


How to configure which GPU to use

there is no config parameter to configure which GPU to use,it always to find GPU:0
I want to change the GPU number,what should i do ?
PS : i used os.environ["CUDA_VISIBLE_DEVICES"] = "3" but it also to run on GPU:0

Try using NVIDIA_VISIBLE_DEVICES.

I also need a "device_map" argv to config this like HF.
Not sure why there is no such argv...

You must set CUDA_VISIBLE_DEVICES=3 outside your script, setting it inside might be too late.


Make spin emote = resting optional

Title. Also maybe allow to choose which emote you would like to replace, IF you would even want that in the first place.

I'll make it optional for next update

Function added


[sensor_ctrl] Add IO status change as an interrupt

First seven commits will be rebased away.
Dependent on #12074 and #12079

LGTM. I didn't know that plic unittest is now autogenerated :) One thing less to work when interrupts are revised.

actually is it auto-gen'd? because i completely forgot about it and CI just failed heh

LGTM. I didn't know that plic unittest is now autogenerated :) One thing less to work when interrupts are revised.

actually is it auto-gen'd? because i completely forgot about it and CI just failed heh

Do you mean sw/device/lib/dif/dif_rv_plic_unittest.cc or sw/device/tests/autogen/plic_all_irqs_test.c? Because the latter is the only that is autogen'd IIRC.


cd Command doesn't work

If you type in 'cd' and the folder name, it stays in the /bin folder

Fixed in release v0.0.4


chore: make name display more concise on the stack

This PR is related to issue #373 , which expecting to make name display more concise on the stack

Hi! Thanks for taking a look at this. This issue is quite old now, and I agree it doesn't generalize across profile format types. I'm going to close this for now.


RFC: single "base class" for uv_handle_t and uv_req_t

I would like to have a single base type for handles and requests, and the uv_handle_type and uv_req_type types should share a name space.
The reason for this move is that it'll allow me to put requests and handles in the same queue, so I can get rid of a separate "endgame queue". That will quite easily let unix and windows share the same event loop core.
typedef struct {
  uv_object_type_t type;
  void* queue[2];
   void* data;
} uv_object_t;

Instead will the fields be contained within a macro, much like it is now?

Sounds like a good idea to me!
On Jan 2, 2015 8:28 PM, "Bert Belder"<EMAIL_ADDRESS>wrote:

I would like to have a single base type for handles and requests, and the
uv_handle_type and uv_req_type types should share a name space.
The reason for this move is that it'll allow me to put requests and
handles in the same queue, so I can get rid of a separate "endgame queue".
That will quite easily let unix and windows share the same event loop core.
typedef struct {
uv_object_type_t type;
void* queue[2];
void* data;
} uv_object_t;
—
Reply to this email directly or view it on GitHub
https://github.com/libuv/libuv/issues/102.

No objections, as long as I don't have to do the work. :-)

I was thinking of https://github.com/libuv/libuv/commit/91a702aab1ea2e24aaee6c77a2764e74cc8fe846#diff-6

Total nit, but is "object" the best word choice? This is coming from the
massively confused JS side of me. :-P

I was thinking of 91a702a#diff-6

I looked at the whole diff and it looks good!

Total nit, but is "object" the best word choice? This is coming from the massively confused JS side of me. :-P

In a way we are emulating some OO hierarchy with that, so find "object" appropriate here IMHO.

uv_guess_handle still has the word handle in it, but i can't come up with a better name

Continuing in #116


Create an e2e test for pod without service

As a follow-up to https://github.com/openservicemesh/osm/pull/1932 - create an end-to-end test to ensure that a pod without a dedicated Kubernetes Service can cURL bookstore.

Scope (please mark with X where applicable)

New Functionality          [ ]
Install                    [ ]
SMI Traffic Access Policy  [ ]
SMI Traffic Specs Policy   [ ]
SMI Traffic Split Policy   [ ]
Permissive Traffic Policy  [ ]
Ingress                    [ ]
Egress                     [ ]
Envoy Control Plane        [ ]
CLI Tool                   [ ]
Metrics                    [ ]
Certificate Management     [ ]
Sidecar Injection          [ ]
Logging                    [ ]
Debugging                  [ ]
Tests                      [X]
CI System                  [X]
Project Release            [ ]

Possible use cases

@nojnhuh had already fixed this, and we already have a test for it if not mistaken (e2e_pod_client_server_test.go)

@eduser that test still creates a service on the client side. I hadn't made this change yet since I knew it was going to fail without the service. It would be like a 3 line change to get rid of it though.


how to have more than one home

Subject: how to have more than one home
Description:
https://couchers.org/profile/edit/about
hello
i need to be able to have two different homes listed
thanks
Results:
Backend version: develop-7c33e18d
Frontend version:
User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36
Page: https://couchers.org/profile/edit/about
User: calicia (13391)

not a bug


Vulkan pixel history fails to detect shader discards on Intel

Description
Pixel history does not properly detect shader discards (presumably only on Intel drivers). For discards, it seems like shader out shows as all black, while tex after is unchanged. (Though, shader out might simply be undefined and in my case it just happens to be black).
Steps to reproduce

This can be reproduced by running the VK_Pixel_History functional test.

$ ./renderdoccmd test functional -t VK_Pixel_History
...
>> Test VK_Pixel_History
    .. Running exe:'C:\Users\w.pearson\source\repos\renderdoc-contrib\util\test\demos_x64.exe' cmd:'VK_Pixel_History --log C:\Users\w.pearson\source\repos\renderdoc-contrib\util\test\tmp\demos.log' in dir:'' with env:'[]'
main.cpp:678 Log: Running 'VK_Pixel_History'
    .. Queuing capture of frame 5..6 with timeout of default
vk\vk_test.cpp:294 Log: Initialising Vulkan at VK1.3
vk\vk_test.cpp:750 Log: Running Vulkan test on Intel(R) UHD Graphics (version 1.3)
    >> Raw Process output
        vk\vk_test.cpp:294 Log: Initialising Vulkan at VK1.3
        vk\vk_test.cpp:750 Log: Running Vulkan test on Intel(R) UHD Graphics (version 1.3)
    << Raw Process output
    .. Loading capture
    .. Checking capture
    .. Testing pixel 190, 149
    ** Picked value at 190,149 in Swapchain Image 176 is as expected
    .. Testing pixel 190, 150
    ** Picked value at 190,150 in Swapchain Image 176 is as expected
    .. Testing pixel 200, 50
    ** Picked value at 200,50 in Swapchain Image 176 is as expected
    .. Testing pixel 150, 250
    !+ FAILURE in VK_Pixel_History: TestFailureException eventId 21, primitiveID 0: testing shader_discarded expected True, got False
    >> Callstack
        File "tests/Vulkan/VK_Pixel_History.py", line 397, in check_events
            actual))
        File "tests/Vulkan/VK_Pixel_History.py", line 122, in primary_test
            self.check_events(events, modifs, False)
        File "tests/Vulkan/VK_Pixel_History.py", line 37, in check_capture
            self.primary_test()
        File "rdtest/testcase.py", line 493, in run
            self.check_capture()
        File "rdtest/testcase.py", line 500, in invoketest
            self.run()
        File "rdtest/runner.py", line 349, in do
            instance.invoketest(debugMode)
        File "rdtest/runner.py", line 357, in run_tests
            do(False)
    << Callstack
    !- FAILURE
    $$ FAILED
<< Test VK_Pixel_History
...

The test passes if the check_events call on line 122 is commented out (i.e. everything else behaves correctly; it's only discards that are broken).
In RenderDoc itself, this can be seen just by using pixel history at x=150 and observing that e.g. EID 21 is not shown as a discard. (That's how I checked the stable version).
Environment

RenderDoc version: v1.26, self-built eb643dd76f92aed18f9837fe942d5491feb8f3e1. Demos from eb643dd76f92aed18f9837fe942d5491feb8f3e1.
Operating System: Windows 10.0.19042.2728
Graphics API: Vulkan
CPU: 12th Gen Intel(R) Core(TM) i7-1270P, 2200 Mhz, 12 Core(s), 16 Logical Processor(s)
GPU: Intel(R) UHD Graphics, driver version <IP_ADDRESS>55

This looks to be a different manifestation of a driver bug previously encountered. I've expanded the workaround that seems to work for me, but it may or may not work depending on what the driver bug is.

Thanks. That workaround seems to at least fix the broken test on my machine. If I find any other broken edge-cases I'll open a new issue.

At this point I would suggest that you don't unless it repro's on a non-Intel GPU. Intel seems quite broken.

I was looking over this code again and I noticed that there seems to be a mistake:
https://github.com/baldurk/renderdoc/blob/1a2db41fa807dd0bdc4afe67f5d92590fef9dee7/renderdoc/driver/vulkan/vk_pixelhistory.cpp#L785-L788
Shouldn't the second line use dst instead of src?
atts[i].dstColorBlendFactor = atts[i].dstAlphaBlendFactor = VK_BLEND_FACTOR_ONE;

I tested it out locally and it seems like both versions behave correctly on my machine. (I'm not sure if that makes sense or not.)
(As a minor side thing, atts[i].colorWriteMask = VK_COLOR_COMPONENT_A_BIT would probably also be clearer than using 0x8, but that shouldn't matter either.)


[WIP] add blob_announce ability to specify lists of blobs

need to announce multiple blobs at once on reflector since they don't keep files

only for temporary use closing now


Not working with Rematch 5 beta

Since this will be the new version can we get an update that these work together again. Without scripts is like life without coffee

Relevant information for us from Gello: https://www.warcraftpets.com/community/forum/viewtopic.php?p=149251&sid=87e17bc5c4b93d1dfbc356c918f00e7d#p149251
They really don't want us to store script information in the team data, but are offering to do API for extending Rematch. I didn't have time to look into it since sadly.

So, I've gotten a working edit. It is not backwards compatible, at all (that can be done, but will still leave the file even more messy than it already is)
One major issue though...
In Rematch4 teams are saved with a target ID (number) or a team name (string). As such PBS does the same. On the initial load of Rematch5 those teams are converted to a new format which uses a a unique id (string). It should be possible to convert those over, and have them matchup. Unless if the user been using Rematch5 without PBS, and have made changes to their teams, like myself.

I can't wait to try it out. I have and will never pet battle without this mod. Thanks for all the hard work and dedication

I don't mean to take over development on this, but I really wanted to get access to my scripts again. You can see my branch here. I'm sure the update script could use more testing. And maybe making the code backwards compatible (if thats something @axc450 wants)

So, I've gotten a working edit. It is not backwards compatible, at all (that can be done, but will still leave the file even more messy than it already is)
One major issue though...
In Rematch4 teams are saved with a target ID (number) or a team name (string). As such PBS does the same. On the initial load of Rematch5 those teams are converted to a new format which uses a a unique id (string). It should be possible to convert those over, and have them matchup. Unless if the user been using Rematch5 without PBS, and have made changes to their teams, like myself.

Does this mean you have a version that works with Rematch 5?

Does this mean you have a version that works with Rematch 5?

Technically yes it does work with with Rematch5, but there are some carveouts with it, mainly around the script database migration.

First is it is one way, and will require editing files to go back to Rematch4.
Same goes for starting over and reimporting from Rematch4, if you were to "reset everything" in Rematch5.
Next is I've only tested it on my own scripts. So it may still be buggy, so please backup your saved vars. But it seems to have worked well,
Last and most of all, if you've been using Rematch5 and have made changes to your teams names, or first teams for a target. Scripts will not lineup, and there's no real way to make them.

Technically yes it does work with with Rematch5, but there are some carveouts with it, mainly around the script database migration.

If I backed up my saved variables and then basically reloaded all the teams does that sound like a good way to approach it?
Also - it that patched version linked here?
Having a crush week at work and a bit brain broken - so being really basic about my assumptions.

Technically yes it does work with with Rematch5, but there are some carveouts with it, mainly around the script database migration.

If I backed up my saved variables and then basically reloaded all the teams does that sound like a good way to approach it?
Also - it that patched version linked here?
Having a crush week at work and a bit brain broken - so being really basic about my assumptions.

You have to change the code yourself from his changes to the file or I am GitHub dumb. They work great though. Had to redo the changes when the mod updated last week.

If I backed up my saved variables and then basically reloaded all the teams does that sound like a good way to approach it?

The initial load needs to happen either at the same time, or after Rematch5.

If you've used Rematch5, do a /rematch reset everything
Logout and apply my changes
Log into the game and pray things work.

Also - it that patched version linked here?

You can find my branch Here. Download the Github Zip. Looks like you can copy the contents of the folder that contains the tdBattlePetScript.toc file. The only changes were done to the Rematch folder. You could probably only copy that.

after today's most recent Rematch beta I get the following errors:
1x Interface\FrameXML\Bindings.xml:1 tdBattlePetScript/Rematch/Rematch.xml:1 tdBattlePetScript/Rematch/Rematch.xml(17): error: not well-formed (invalid token)
1x tdBattlePetScript/Rematch/Rematch.xml:1 tdBattlePetScript/Rematch/Rematch.xml(17): error: not well-formed (invalid token)

That's really weird. Not happening for me. Have tried restarting the client? My edit didn't do any change to the Rematch.xml file that would warrant any kind of error. Seems like one of those weird blizzard bugs.

I'll try restarting in a sec - I disables PBS in the addon list before loading in the first time with the new version of rematch, did a rematch reset everything, turned PBS back on - got that error
I have tried reloading a few times but not restarting the client

Uh, if you did a /rematch reset everything you're going to have to reset your tdBattlePetScript.lua SavedVariables file. As the team IDs wont lineup anymore.
Either copy in your backup, or if your comfortable editing the file: delete the ["Rematch"] = { table, and rename the ["Rematch4"] = { to "Rematch"

I have both restarted the client and deleting the tdBattlePetScript.lua from both the account and character level (I had a back up) and I am still getting the same error

With Rematch beta-15, guessing and hoping for matching teams is no longer needed. My branch has been updated to use that.
It should* now reset the scripts back to the Rematch4 backup, if you call /rematch reset everything .
(I did have one issue during testing where I lost all my scripts. So PLEASE MAINTAIN BACKUPS)

Please let me know if we need anything asked/changed on the Rematch end to facilitate this.


Use overlayLayer and overlayMouseTarget

Currently, the lib is drawing the visible target on the overlayMouseTarget layer instead of the overlayLayer. My understanding of how these layers are supposed to work is: visuals on overlay layer, synced, invisible eventTargets on the overlayMouseTarget layer.
I came to this conclusion by picking apart the default map markers and this documentation: https://developers.google.com/maps/documentation/javascript/reference/overlay-view#MapPanes
With this setup, the z-index logic can remain simple and the clusters will continue to show up over pins but under other UI.
fixes #506

@googlebot I signed it!

@googlebot I signed it!

I noticed that there's a big ES6 rewrite happening. Is this fix still necessary?

@ashleahhill I'm not sure when I will get to rewriting this one. I'll defer to @garylittleRLP for the comments already made here.

I'm also fine with rebasing this off of the 3.0 work or opening another PR for 3.0.

closing in favor of #558

@ashleahhill see https://github.com/googlemaps/v3-utility-library/pull/558#issuecomment-547686110 to consent to your commit being merged


Fix double arrows on dropdown on audio settings

STR:

go to any chat
go to audio settings
double arrows appear

macOS

Hi Sara 👋
The problem seems to be fixed, can you confirm?

🎉


PutioAPI.Files.Upload 401 but PutioAPI.Account.Info() works

Hi, I do not understand what I am doing wrong. I initialize the API client like this
      const putioConfig = this.configService.get('putio');
      this.api = new PutioAPI({
        clientID: putioConfig.clientId,
      });
      this.api.setToken(putioConfig.oAuthToken);

with this API when I call the Account.Info() it works perfectly as expected.
  const accountInfo = await putio.Account.Info();
  this.logger.log(`Uploading as ${accountInfo.data.info.username}`);

However when I call the Files.Upload(...) I get 401
      await api.Files.Upload({
        file: absolutePath,
        fileName: filename,
        parentId: targetFolderId,
      });

is there some issue with the sdk or am I doing something wrong?

hey @ptheofan it's on me... 😞 working on a fix now!

should be fixed in v8.35.0, sorry for the late response...


"Number must be greater than 0" Errors

Hello! I'm likely doing something wrong here, but, trying to work through a sample application that is processing a single small document and then querying the index for information. I'm attempting to do this all locally using Ollama - but I have not been able to to get past this set of errors.
Any ideas or suggestions on what to try?
Number must be greater than 0 (["chunkSize"])
Number must be greater than or equal to 0 (["chunkOverlap"])
Chunk overlap must be less than chunk size. ([])

Running this in Node v22.9.0.
Example code:
import {
    Settings,
    Ollama,
    VectorStoreIndex,
    SimpleDirectoryReader,
    OllamaEmbedding,
} from 'llamaindex'

const MODEL_NAME = 'llama3.2'
const ollama = new Ollama({
    model: MODEL_NAME,
    config: {},
    options: { temperature: 0.5 },
})
const embedModel = new OllamaEmbedding({
    model: 'mxbai-embed-large',
    options: { temperature: 0.5 },
})

Settings.llm = ollama
Settings.embedModel = embedModel

async function main() {
    const dataDocs = await new SimpleDirectoryReader().loadData({
        directoryPath: './data',
    })

    console.log(`Loaded ${dataDocs.length} doument(s).`)

    const index = await VectorStoreIndex.fromDocuments(dataDocs, {
        logProgress: true,
    })
    const retriever = index.asRetriever()
    const queryEngine = index.asQueryEngine({
        retriever: retriever,
    })
    const stream = await queryEngine.query({
        query: '{{Question}}',
        stream: true,
    })

    for await (const chunk of stream) {
        process.stdout.write(chunk.delta)
    }
}
main().catch((e) => console.error(e))

Using node parser on documents...
Finished parsing documents.
getting embedding progress: 0 / 1
ZodError: [
  {
    "code": "too_small",
    "minimum": 0,
    "type": "number",
    "inclusive": false,
    "exact": false,
    "message": "Number must be greater than 0",
    "path": [
      "chunkSize"
    ]
  },
  {
    "code": "too_small",
    "minimum": 0,
    "type": "number",
    "inclusive": true,
    "exact": false,
    "message": "Number must be greater than or equal to 0",
    "path": [
      "chunkOverlap"
    ]
  },
  {
    "code": "custom",
    "message": "Chunk overlap must be less than chunk size.",
    "path": []
  }
]
    at get error (file:///./node_modules/.pnpm/zod@3.23.8/node_modules/zod/lib/index.mjs:587:31)
    at ZodEffects.parse (file:///./node_modules/.pnpm/zod@3.23.8/node_modules/zod/lib/index.mjs:692:22)
    at new SentenceSplitter (file:///./node_modules/.pnpm/@llamaindex+core@0.2.8_@aws-crypto+sha256-js@5.2.0_@xenova+transformers@2.17.2_js-tiktoken@1._bmi77gdnidaxf3lqzmibpj7squ/node_modules/@llamaindex/core/node-parser/dist/index.js:318:57)
    at PromptHelper.getTextSplitterGivenPrompt (file:///./node_modules/.pnpm/@llamaindex+core@0.2.8_@aws-crypto+sha256-js@5.2.0_@xenova+transformers@2.17.2_js-tiktoken@1._bmi77gdnidaxf3lqzmibpj7squ/node_modules/@llamaindex/core/indices/dist/index.js:72:16)
    at PromptHelper.repack (file:///./node_modules/.pnpm/@llamaindex+core@0.2.8_@aws-crypto+sha256-js@5.2.0_@xenova+transformers@2.17.2_js-tiktoken@1._bmi77gdnidaxf3lqzmibpj7squ/node_modules/@llamaindex/core/indices/dist/index.js:82:35)
    at CompactAndRefine.getResponse (file:///./node_modules/.pnpm/@llamaindex+core@0.2.8_@aws-crypto+sha256-js@5.2.0_@xenova+transformers@2.17.2_js-tiktoken@1._bmi77gdnidaxf3lqzmibpj7squ/node_modules/@llamaindex/core/response-synthesizers/dist/index.js:209:44)
    at CompactAndRefine.synthesize (file:///./node_modules/.pnpm/@llamaindex+core@0.2.8_@aws-crypto+sha256-js@5.2.0_@xenova+transformers@2.17.2_js-tiktoken@1._bmi77gdnidaxf3lqzmibpj7squ/node_modules/@llamaindex/core/response-synthesizers/dist/index.js:31:35)
    at RetrieverQueryEngine._query (file:///./node_modules/.pnpm/llamaindex@0.6.13_@aws-sdk+client-sso-oidc@3.662.0_@aws-sdk+client-sts@3.662.0__@aws-sdk+cred_ottdx7krjsm3z4s7c6pq2277m4/node_modules/llamaindex/dist/engines/query/RetrieverQueryEngine.js:15:49)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async RetrieverQueryEngine.query (file:///./node_modules/.pnpm/@llamaindex+core@0.2.8_@aws-crypto+sha256-js@5.2.0_@xenova+transformers@2.17.2_js-tiktoken@1._bmi77gdnidaxf3lqzmibpj7squ/node_modules/@llamaindex/core/query-engine/dist/index.js:405:26)
    at async main (C:\Depot\AI\mtg-processor\index.ts:69:20) {
  issues: [
    {
      code: 'too_small',
      minimum: 0,
      type: 'number',
      inclusive: false,
      exact: false,
      message: 'Number must be greater than 0',
      path: [Array]
    },
    {
      code: 'too_small',
      minimum: 0,
      type: 'number',
      inclusive: true,
      exact: false,
      message: 'Number must be greater than or equal to 0',
      path: [Array]
    },
    {
      code: 'custom',
      message: 'Chunk overlap must be less than chunk size.',
      path: []
    }
  ],
  addIssue: [Function (anonymous)],
  addIssues: [Function (anonymous)],
  errors: [
    {
      code: 'too_small',
      minimum: 0,
      type: 'number',
      inclusive: false,
      exact: false,
      message: 'Number must be greater than 0',
      path: [Array]
    },
    {
      code: 'too_small',
      minimum: 0,
      type: 'number',
      inclusive: true,
      exact: false,
      message: 'Number must be greater than or equal to 0',
      path: [Array]
    },
    {
      code: 'custom',
      message: 'Chunk overlap must be less than chunk size.',
      path: []
    }
  ]
}

This is werid, I can run it successfully. Make sure ./data directory load document successfully

Oh, i see the error when there's only one document

I can see the problem here is context size is even smaller than prompt length size

Please set num_ctx to the correct value, because we cannot help you to get the correct llm metadata
const ollama = new Ollama({
	model: MODEL_NAME,
	config: {},
	options: {
		temperature: 0.5,
	  num_ctx: 128_000,
	},
})

I seem to get the same error with more than one document and longer documents. It seems like the chunk size and chunk overlap are being seen as zero (or not set), but I'm not sure why.

Please set num_ctx to the correct value, because we cannot help you to get the correct llm metadata

I set this, and it gives the same results.

I set this, and it gives the same results.

So the reason is from loader side, you might load a too large content into one Document.
You can run text spliiter after load the document
const dataDocs = await new SimpleDirectoryReader().loadData({
  directoryPath: './empty',
})
const textSplitter = new SentenceSplitter()
const docs = await textSplitter(dataDocs)

So the reason is from loader side, you might load a too large content into one Document.

This document is short (4kb). The loader code seems to run fine but fails after it requests embeddings (though embeddings return manually if i pass the content to Ollama manually via API.

Updated to llamaindex v0.6.15 and still encounter the same errors.

I set this up to store the index to disk and the system is generating what appears to be valid vector stores, etc. doc_store.json, index_store.json, and vector_store.json all get populated. But running an actual query still fails with the same errors.

From what I've been able to figure out so far is it appears there is a SentenceSplitter getting setup somewhere that is missing or has bad configuration... But I'm not specifically sure where that is happening inside of the query chain. But calling query on an index fails before making a call to the LLM - so it seems like an internal Llamaindex configuration issue.

I'm also having the same issue with Ollama.  My goal is to use Ollama and local embeddings instead of OpenAI
Code:
import fs from "node:fs/promises";

import {
  Document,
  MetadataMode,
  NodeWithScore,
  VectorStoreIndex,
  Ollama,
  OllamaEmbedding,
  Settings,
  HuggingFaceEmbedding,
  OpenAI
} from "llamaindex";

const MODEL_NAME = 'llama3';

const ollama = new Ollama({ model: "llama3" });
const embedModel = new HuggingFaceEmbedding({
  modelType: "BAAI/bge-small-en-v1.5",
  quantized: false,
});

Settings.llm = ollama;
Settings.embedModel = embedModel;

async function main() {
  // Load essay from abramov.txt in Node
  const path = "node_modules/llamaindex/examples/abramov.txt";

  const essay = await fs.readFile(path, "utf-8");

  // Create Document object with essay
  const document = new Document({ text: essay, id_: path });

  // Split text and create embeddings. Store them in a VectorStoreIndex
  const index = await VectorStoreIndex.fromDocuments([document]);

  // Query the index
  const queryEngine = index.asQueryEngine();
  const { response, sourceNodes } = await queryEngine.query({
    query: "What did the author do in college?",
  });

  // Output response with sources
  console.log(response);

  if (sourceNodes) {
    sourceNodes.forEach((source: NodeWithScore, index: number) => {
      console.log(
        `\n${index}: Score: ${source.score} - ${source.node.getContent(MetadataMode.NONE).substring(0, 50)}...\n`,
      );
    });
  }
}

main().catch(console.error);

I get this error
(node:38005) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
ZodError: [
  {
    "code": "too_small",
    "minimum": 0,
    "type": "number",
    "inclusive": false,
    "exact": false,
    "message": "Number must be greater than 0",
    "path": [
      "chunkSize"
    ]
  },
  {
    "code": "too_small",
    "minimum": 0,
    "type": "number",
    "inclusive": true,
    "exact": false,
    "message": "Number must be greater than or equal to 0",
    "path": [
      "chunkOverlap"
    ]
  },
  {
    "code": "custom",
    "message": "Chunk overlap must be less than chunk size.",
    "path": []
  }
]
    at get error (LlamaIndexTS/node_modules/zod/lib/types.js:55:31)
    at ZodEffects.parse (LlamaIndexTS/node_modules/zod/lib/types.js:160:22)
    at new SentenceSplitter (LlamaIndexTS/node_modules/@llamaindex/core/node-parser/dist/index.cjs:320:67)
    at PromptHelper.getTextSplitterGivenPrompt (LlamaIndexTS/node_modules/@llamaindex/core/indices/dist/index.cjs:74:16)
    at PromptHelper.repack (LlamaIndexTS/node_modules/@llamaindex/core/indices/dist/index.cjs:84:35)
    at CompactAndRefine.getResponse (LlamaIndexTS/node_modules/@llamaindex/core/response-synthesizers/dist/index.cjs:211:44)
    at CompactAndRefine.synthesize (LlamaIndexTS/node_modules/@llamaindex/core/response-synthesizers/dist/index.cjs:33:35)
    at RetrieverQueryEngine._query (LlamaIndexTS/node_modules/llamaindex/dist/cjs/engines/query/RetrieverQueryEngine.js:30:45)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async RetrieverQueryEngine.query (LlamaIndexTS/node_modules/@llamaindex/core/query-engine/dist/index.cjs:407:26)
    at main (LlamaIndexTS/main.ts:54:37) {
  issues: [
    {
      code: 'too_small',
      minimum: 0,
      type: 'number',
      inclusive: false,
      exact: false,
      message: 'Number must be greater than 0',
      path: [Array]
    },
    {
      code: 'too_small',
      minimum: 0,
      type: 'number',
      inclusive: true,
      exact: false,
      message: 'Number must be greater than or equal to 0',
      path: [Array]
    },
    {
      code: 'custom',
      message: 'Chunk overlap must be less than chunk size.',
      path: []
    }
  ],
  addIssue: [Function (anonymous)],
  addIssues: [Function (anonymous)],
  errors: [
    {
      code: 'too_small',
      minimum: 0,
      type: 'number',
      inclusive: false,
      exact: false,
      message: 'Number must be greater than 0',
      path: [Array]
    },
    {
      code: 'too_small',
      minimum: 0,
      type: 'number',
      inclusive: true,
      exact: false,
      message: 'Number must be greater than or equal to 0',
      path: [Array]
    },
    {
      code: 'custom',
      message: 'Chunk overlap must be less than chunk size.',
      path: []
    }
  ]
}

Everything works fine when I use OpenAI as the model instead an Ollama model
const openai = new OpenAI({ model: "gpt-3.5-turbo", temperature: 0 });

const embedModel = new HuggingFaceEmbedding({
  modelType: "BAAI/bge-small-en-v1.5",
  quantized: false,
});

Settings.llm = openai
Settings.embedModel = embedModel

working on it


Knowledge Graph Application. Link prediction error

Hi,
I was trying to train KG embedding and perform link prediction with a custom dataset. When I run
predictions = app.entity_prediction(file_name='valid.txt', target="tail", k=5)
I get error at this line:
assert len(new_R) == len(R), "Can't recognize some entities or relations"
I have been reading the code for a while but can't figure it out. Any idea what might cause this?
BTW, sometime I get loss=nan during training, after the 5th iteration, especially when using small dimensions. Don't know why.
Thank you for the help!

This indicates that some relations in the test set have never been seen in the training set, so the tail entities of these triplets can't be predicted. You may filter these triplets in your test set.
loss=nan is usually a result of large learning rate. For knowledge graph embedding, it is necessary to tune both lr and relation_lr_multiplier. See the config files for Wikidata5m to have an idea of these hyperparameters on large-scale datasets.

Hi,
Thanks for your reply.
I checked the data I used. All the relations in test set appears in the training set. I have only 2 relations, GoodFor and BadFor.

Maybe some entities in the test set don't occur in the training set?

I see. That may be the problem. I was expecting to have error when checking the length of the entities. Let me try what you suggested and see if it works.
Thank you very much!

I see. That may be the problem. I was expecting to have error when checking the length of the entities. Let me try what you suggested and see if it works.
Thank you very much!

The problem is some entities in test set does not appear in training set. It's solved now. Thank you!

This indicates that some relations in the test set have never been seen in the training set, so the tail entities of these triplets can't be predicted. You may filter these triplets in your test set.
loss=nan is usually a result of large learning rate. For knowledge graph embedding, it is necessary to tune both lr and relation_lr_multiplier. See the config files for Wikidata5m to have an idea of these hyperparameters on large-scale datasets.

Maybe some entities in the test set don't occur in the training set?

英文不行，我还是用中文吧。我也是遇到了这个问题，你提到说是test中的一些实体没在train中出现过，所以会报这个错误，但是我看了看FB15k-237数据集，test也有几个（10348-10319=29）几个train不认识的实体，我很好奇这个数据集是否能通过测试。
另外，我想请教下application.name_map这个函数究竟是在解决什么问题？


Add update Oracle price functionality to Honey Pot

It should say "Oracle price" with a tooltip explaining "This is the price used for funding proposals defined in xdai" the price should be below or close by and a button "Update"/"Update price" or whatever you think which creates a transaction. It will be greyed out when it doesn't need to be clicked (it only needs to be clicked once every 3 hours).

Here are the designs: https://www.figma.com/file/P06HTzSyb9hExJVpwaUPmA/Honey-Pot-2.0?node-id=3146%3A18562

Thanks @fioreb


code fences issues

hi, when I use the code fences ,in the raw mode ,after that every raw will be fences.like this:

Just pushed a commit (f8b1817a5ae9c59af6607d9daec0d1e39de93ba1) to fix this. I should be able to get a new release up and ready tomorrow or the day after

Release is up, so this should resolve the issue~

thanks:)


Sending backdated telemetry to ApplicationInsights

.NET 5 Console App
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net5.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.ApplicationInsights.WorkerService" Version="2.21.0" />
  </ItemGroup>
</Project>

Describe the bug
Backdated telemetry to ApplicationInsights is not recorded .
To Reproduce
When timestamp is on current day the telemetry is record, any backed trace entries are not recorded.
 foreach ( var backDate in Enumerable.Range( 1, 100 ) )
      {
        var timeStamp = DateTimeOffset.UtcNow.AddDays(-backDate);
        Console.WriteLine($"BackDated event occurred on {timeStamp}");

        var telemetry1 = new TraceTelemetry( $"BackDated event occurred on {timeStamp} with event timestamp", SeverityLevel.Information );
        telemetry1.Timestamp = timeStamp;
        telemetry1.Properties.Add( "ActualEventDate", timeStamp.ToString() );
        TelemetryClientInstance.TrackTrace( telemetry1 );

        var telemetry2 = new TraceTelemetry( $"BackDated event occurred on {timeStamp} with current timestamp ", SeverityLevel.Information );
        telemetry2.Timestamp = DateTimeOffset.UtcNow;
        telemetry2.Properties.Add( "ActualEventDate", timeStamp.ToString() );
        TelemetryClientInstance.TrackTrace( telemetry2 );
      }

      Console.WriteLine( "Waiting for 5 secs to flush TelemetryClient." );
      await Task.Delay( 5000 );
      TelemetryClientInstance.Flush();

Telemetry Payload with event timestamp
Application Insights Telemetry: {
    "name": "AppTraces",
    "time": "2022-11-04T11:31:01.2050453Z",
    "iKey": "--removed--",
    "tags": {
        "ai.cloud.roleInstance": "SRV-9961",
        "ai.internal.sdkVersion": "dotnetc:2.21.0-429"
    },
    "data": {
        "baseType": "MessageData",
        "baseData": {
            "ver": 2,
            "message": "BackDated event occurred on 04/11/2022 11:31:01 +00:00 with event timestamp",
            "severityLevel": "Information",
            "properties": {
                "ActualEventDate": "04/11/2022 11:31:01 +00:00"
            }
        }
    }
}

Telemetry Payload with current timestamp
Application Insights Telemetry: {
    "name": "AppTraces",
    "time": "2023-02-08T11:31:01.2102101Z",
    "iKey": "--removed--",
    "tags": {
        "ai.cloud.roleInstance": "SRV-9961",
        "ai.internal.sdkVersion": "dotnetc:2.21.0-429"
    },
    "data": {
        "baseType": "MessageData",
        "baseData": {
            "ver": 2,
            "message": "BackDated event occurred on 04/11/2022 11:31:01 +00:00 with current timestamp",
            "severityLevel": "Information",
            "properties": {
                "ActualEventDate": "04/11/2022 11:31:01 +00:00"
            }
        }
    }
}

Entire telemetry 100 entries with the current timestamp are recorded and viewable in Application Insights.
traces
| where message contains "BackDated event occurred"
| order  by timestamp asc

Not an SDK bug. Backend does not allow older items (48 or so).
Even within SDK items older than 48hours gets dropped by ServerTelemetryChannel : https://learn.microsoft.com/en-us/azure/azure-monitor/app/telemetry-channels#does-the-application-insights-channel-guarantee-telemetry-delivery-if-not-what-are-the-scenarios-in-which-telemetry-can-be-lost


Add coverage tool

See https://github.com/DAI-Lab/cookiecutter-pypackage/issues/1

Closed by bf949e14cd9559a379215d271f66d60949e07fc6


Review integration digest computation and projection to the integration container as env var

What happened?
Since an integration digest ends up begin projected as a container env var, including the traits when computing the digest has a number of side effects, as example by enabling/disabling the creation of a route or service, would lead to the container to be restarted which is not expected.
Steps to reproduce
No response
Relevant log output
No response
Camel K version
No response

Triggering a reconciliation is fine, but that should not lead to a disruptive change in any resource so maybe we need two different digests or rethink how the reconciliation is re-triggered

If you have an integration like this one (flow omitted because it does not matter much):
apiVersion: camel.apache.org/v1
kind: Integration
metadata:
  name: r
spec:
  flows:
    ...
  traits:
    service:
      enabled: true
      auto: false
    ingress:
      enabled: true
      auto: false

And then:

deploy it
change ingress.enable to false
deploy it

Then what happen is that the pod is restarted
➜ k get pods
NAME                                READY   STATUS        RESTARTS   AGE
camel-k-operator-6897f796dd-5znkm   1/1     Running       0          22m
r-677dcbf7fc-h7wx6                  0/1     Terminating   0          22s
r-6976c6b54b-8lccc                  1/1     Running       0          2s

Then, ignoring some of the stuffs that are changing as as side effect of pod restart (uuid, dates, etc), the only difference is in the digest:

Okey. Indeed, it's because the env template vars in pod changes. I've checked and it seems this var is not really used anywhere. I'll try to remove it at all and run some test to see if it's good to solve this problem.

The CAMEL_K_DIGEST env var was added to force the re-deploy of the pod in case of a config map changes, like if only change the flow in an Integration, then the pod won't actually change, only the config map will so we had to add this variable to be sure the latest routes are taken into account

I wonder if we have to store them at all:

the one for the pod is an ephemeral value that depends on the content of an integration and some external resources, I don't think there is any value in storing it in the status also because we are computing it for every reconciliation (at least, this is my understanding) and we can eventually compute it only when needed (i.e. we don't need to generate it for a number of transient phases)
we can optionally compute the digest of the previous and current object in the event filter (i.e. when the generation or phase have changed we don't have to compute it)
storing the checksum in the status may trigger additional spurious reconcile loops

Not sure this would cover all the cases, WDYT ?

* we can optionally compute the digest of the previous and current object in the event filter (i.e. when the generation or phase have changed we don't have to compute it)

This one was an option I've evaluated but we need to discard because we have other objects beside the Integration. Effectively, the main problem as I mentioned above is that we watch a lot of other resources beside the Integration, so, the only possible way to know the status before, is to have this information stored in the same resource for eventual comparison. When an Integration spec change we have a certain degree of control, but when the change happen in any resource bound to the Integration (ie, the Deployment), then, we need to recover such value and make a comparison. Alternatively we may recalculate from scratch every time as I commented previously: "[...] we could reconcile every time an Integration change as the regeneration of resources belonging to the Integration is idempotent (either the same digest would be the same). However we watch a lot of other objects and the monitoring action could be called so many time degrading the operator performance."
I am not able to forecast such a degradation though.

* we can optionally compute the digest of the previous and current object in the event filter (i.e. when the generation or phase have changed we don't have to compute it)

This one was an option I've evaluated but we need to discard because we have other objects beside the Integration. Effectively, the main problem as I mentioned above is that we watch a lot of other resources beside the Integration, so, the only possible way to know the status before, is to have this information stored in the same resource for eventual comparison. When an Integration spec change we have a certain degree of control, but when the change happen in any resource bound to the Integration (ie, the Deployment), then, we need to recover such value and make a comparison. Alternatively we may recalculate from scratch every time as I commented previously: "[...] we could reconcile every time an Integration change as the regeneration of resources belonging to the Integration is idempotent (either the same digest would be the same). However we watch a lot of other objects and the monitoring action could be called so many time degrading the operator performance."
I am not able to forecast such a degradation though.

is it true that if i.e. a deployment changes then we go to the same filter as the Integration resource  ?  it seems to be odd

is it true that if i.e. a deployment changes then we go to the same filter as the Integration resource ? it seems to be odd

Yes. If you change a value of any Integration managed object, the monitoring loop starts and it applies the trait for the given Integration phase (running). In the case of Deployment it would bring the values back to the ones calculated by the operator.

I know that a reconciliation is triggered but does the integration filter apply ? because if so, then the previous and current integration object would have been the same (i.e. same checksum), so the filter would "skip" the event isn't it ?

Yes, what it would happen is that the Integration would stick in the Running phase, so, only applying the traits expected for such execution. Conversely, if there is a change in the digest, the Integration would start from Initialization phase, passing through all the phases (rebuilding, if the kit is no longer matching the expected configuration). If we remove the digest "optimization", we'd need to run every time this second flow.


Warning The “SIS_DATA” environment variable is not set / or value is not an existing directory

I have dowloaded and run successfully your docker but even after fresh builds with various changes in Dockerfile or added the option  -e "SIS_DATA=/home/vidal/Logiciels/Apache/apache-sis-1.4/data/" I keep at best getting the warning :
[main] WARN org.apache.sis.system {} - The SIS_DATA environment variable is defined, but the given “/opt/fuseki/data” value is not an existing directory.

# even if
ll /opt/fuseki/data
total 12K
drwxr-xr-x 3 root   root   4,0K  7 déc.  16:16 ./
drwxrwxr-x 6 fuseki fuseki 4,0K  7 déc.  16:15 ../
drwxr-xr-x 3 root   root   4,0K  7 déc.  16:16 Databases/

I am not flent with docker nor Java, could you please help me find howyo use SIS resource
Thanks for this very usefull work.

Hi @g-vidal!
This is how you can make it working:

Download Apache SIS (https://sis.apache.org/) ; this seems that you have done this already
Extract the content of the Zip file, and make sure it's mounted somewhere in the Docker image like /apache-sis
Set the following environment variables:

PATH to "/apache-sis/bin:${JAVA_HOME}/bin:${PATH}"
SIS_DATA to "/apache-sis/data"
SIS_OPTS to "--encoding UTF-8"

Download Apache Derby (https://db.apache.org/derby/) and put the jar into /apache-sis/lib (the direct link to the latest version for now is: https://repo1.maven.org/maven2/org/apache/derby/derby/<IP_ADDRESS>/derby-<IP_ADDRESS>.jar)

And then you should be able to start Fuseki without that warning.
If for your use-case you don't have the need for Apache SIS, you can safely ignore this warning.

I did some work to include Apache SIS by default in the Docker image, so that no one get this warning anymore, and don't have to struggle on the way how to integrate it by their own.
I'm closing this issue ; everything is tracked in #31.


[FLINK-17514] Let TaskCancelerWatchDog call TaskManagerActions.notifyFatalError with non-null cause

What is the purpose of the change
This PR hardens the TaskCancelerWatchDog to properly fail the task fatally.
Brief change log

59fee01 Let TaskCancelerWatchDog call TaskManagerActions.notifyFatalError with non-null cause: In order to avoid NPE we have to call TaskManagerActions.notifyFatalError with non-null arguments. This commit changes the behaviour accordingly.

2ae3c56 Fail fatally if the TaskCancelerWatchDog encounters exception in run method: If the TaskCancelerWatchDog encounters an exception in the run method, then we can no longer guarantee that it will do its job. Hence, it is best to fail fatally by letting the exception bubble up so that it is handled by the uncaught exception handler.

9b71bd6 Harden ExceptionUtils.tryEnrichTaskManagerError, .isMetaspaceOutOfMemoryError and .isDirectOutOfMemoryError to handle null values: Error handling code should not fail. Hence, this commit changes the methods ExceptionUtils.tryEnrichTaskManagerError, ExceptionUtils.isMetaspaceOutOfMemoryError and ExceptionUtils.isDirectOutOfMemoryError to properly handle a null argument.

Verifying this change
Modified TaskTest#testFatalErrorAfterUnInterruptibleInvoke and TaskTest#testFatalErrorOnCanceling to test that fatal error cause is non-null.
Added tests to ExceptionUtils which verify that the #tryEnrichTaskManagerError, #isMetaspaceOutOfMemoryError and #isDirectOutOfMemoryError can handle null values.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@TisonKun misclicked your username.

Thanks for the review @GJL. I've addressed your comments. Merging once AZP gives green light.


configure-opsman should support setting custom syslog settings

Extension of #474
extend the configure-opsman command to allow setting custom syslog settings.
Relevant endpoint:  /api/v0/settings/syslog
An example addition to the config file:
syslog-settings:
    syslog-value-key1: value
    syslog-value-key2: value

Requirements:

Should be able to set independently from other endpoints
If top-level key is not provided, does not call the API
update example config in docs and indicate new support https://github.com/pivotal-cf/om/blob/master/docsgenerator/templates/configure-opsman/ADDITIONAL_INFO.md
Docs generator should be run to reflect the docs changes
update changelog

Recommendations:
OpsManager is expecting this JSON payload on the PUT:
{
          "syslog": {
            "enabled": true,
            "address": "<IP_ADDRESS>",
            "port": "514",
            "transport_protocol": "tcp",
            "tls_enabled": true,
            "permitted_peer": "*.example.com",
            "ssl_ca_certificate": "-----BEGIN CERTIFICATE-----\r\nMIIBsjCCARug...",
            "queue_size": 100000,
            "forward_debug_logs": true,
            "custom_rsyslog_configuration": "if $message contains 'test' then stop"
          }
}

Recommend not keeping these values in a struct, but convert the yaml values from the config into a parse-able json object for the API.

this will be included in om 4.7.0


Command to reinstall packages

command to reinstall all packages from requirements.txt and requirements-dev.txt

Added jetzt --reinstall in https://github.com/janikarh/jetzt/commit/7f3443693f479773b42474c7f1af94a8acfb8d45.


chore: fix some comments

Will make changes to separate pr: https://github.com/bluealloy/revm/issues/1348
Thank you for pointing out misspellings


Installing the AWS Toolkit for Visual Studio 2017 Professional causes it to crash immediately

Visual studio was working fine until I got an update yesterday. After the update, Visual Studio would close on it's own about 5 seconds after initializing the app. I uninstalled and reinstalled the app and found that everything was working fine until I went to deploy to Beanstalk this morning. I noticed that I didn't have the AWS Toolkit for Visual Studio 2017 so I installed it. Once I installed this and restarted my machine, Visual Studio displayed the same behavior (immediately closing when opened). After some troubleshooting I found that the error was caused by a malformed credential in my credentials file.
Recommendations
Fail more gracefully. This failure was so nonspecific that I spent a good chunk of time troubleshooting Visual Studio and my operating system because I didn't have a good way of knowing what was causing the issue. There should be some display to the user, and this should NOT crash all of Visual Studio.
Expected Behavior
When I open Visual Studio it should stay open.
Current Behavior
When I open Visual Studio it automatically closes after about 5 seconds or so.
Steps to Reproduce (for bugs)
Explained above.
Context
This has prevented me from working on a portion of my project, and has disabled me from deploying to Beanstalk.
Your Environment
Visual Studio Professional 2017: 15.9.4
AWS Toolkit for Visual Studio 2017: <IP_ADDRESS>
Windows 10 Version	10.0.17134 Build 17134
OS Name	Microsoft Windows 10 Enterprise

Thank you for reporting this @baileyfriend - when you mention malformed credential, do you mean that you had a non-valid key & secret pair, the file format was incorrect (for example, a profile was missing a square bracket), or something else?

I'm trying to install "AWS Toolkit for Visual Studio 2017 community 15.9.4.  Everytime i install via  standalone MSI installer I get an error stating that the Certificate in the package is invalid.  The Nuget version installs only to the solution and that works fine.

      I'm trying to install "AWS Toolkit for Visual Studio 2017 community 15.9.4.  Everytime i install via  standalone MSI installer I get an error stating that the Certificate in the package is invalid.  The Nuget version installs only to the solution and that works fine.

I got it (AWS Toolkit for Visual Studio 2017 community 15.9.4) working fine.  (1) Logged into PC as local admin, (2) Used mobile internet instead of office internet.  The package got installed successfully.


MAINT: Restructure extra_special (working name) as internal library

Reference issue

Closes gh-19499
What does this implement/fix?

This PR creates a new folder for the headers that provide scalar functions in the scipy::special namespace, moves _lambertw.h and its dependency _evalpoly.h there, and creates a private scipy.special._get_include to access this folder that @izaid can use when experimenting with using these functions in CuPy.
_binom.h has not been added yet because it is not properly header only due to its dependence on cephes.
Additional information

cc @izaid, @rgommers

This looks very useful! A couple of suggestions:

Do we want "extra_special" to be the directory, or do we want "scipy/special"?. The question is what the C++ code should look like. I'd prefer #include <scipy/special/lambertw.h> rather than #include <extra_special/lambertw.h> so it's clear what it is.
Regardless of the name, I'd suggest putting it all inside a top "include". So this would look like "include/extra_special" or "include/scipy/special" respectively. This include directory sits in the current special module, aka where we are changing everything now.
_get_include() needs to return what you currently have as "here" in that function. Essentially it should return the location of the "include" above, with "include" but not "extra_special" or "scipy/special".

We'll see where things end up, but I just wanted to get the behavior working with a minimal amount of change. I see extra_special as just a temporary working name at this point.  I've fixed the return value of _get_include.

OK, so my understanding of how this should go, is that extra_special is just being treated as another special functions library that we vendor along with cephes, amos, specfun', etc. This is library for all of the special function compiled kernels that SciPy doesn't take from elsewhere. Eventually, we hope this will become an external library that we include as submodule, as suggested by @rkern in the issue associated to this PR, gh-19515, and by some other commenters in gh-19404. extra_special is just a placeholder name for this library that I chose because I wanted to pick something so bad that it be unlikely to stick.
I no longer think it makes sense to put these functions within the scipy::special namespace, since ideally these namespaces should match the include structure, and there's no intention at this time for SciPy to officially provide this library. I think there should instead be a one level namespace which is for now the temporary working name of the library, either extra_special, or perhaps something else more professional sounding if having to do#include <extra_special/lambertw.h> is too annoying.
At the moment, the goal of this PR is just to provide the headers for internal development purposes to @izaid and anyone else who wants to get onboard with this project, so I think it makes sense to not deform the directory structure of scipy/special to something that might seem odd if there was no consideration of the headers being provided elsewhere.

Following up on my previous comment. I've changed the nested scipy::special namespace to just be extra_special. I also cleaned up usings in lambertw.h which were outside of any namespace, and moved the helper functions in lambert.h to a detail namespace.

If we're going to stick with a single namespace, can we at least just change it to be simply special rather than extra_special? That can be a reasonable placeholder name for some time.

If we're going to stick with a single namespace, can we at least just change it to be simply special rather than extra_special? That can be a reasonable placeholder name for some time.

Yes, definitely.

@steppi Looks good to me! Next up - trying with CuPy.

@steppi There seems to be a few conflicts, can you please fix those? IMO this is ready to merge otherwise!

@steppi There seems to be a few conflicts, can you please fix those? IMO this is ready to merge otherwise!

Sorry. I got codespelled. I didn't expect any of this stuff would get touched. Fixed now.

There's a test failure in special.test_trig.py. trig isn't touched by this PR but is touched in https://github.com/scipy/scipy/pull/19496/files which I'd just worked on, so I had to do a double take. I checked that the failure is appearing in other PRs; not related.


Notification Reminder for Task

The goal of this feature request is to Add Notification Reminder for the task. Current implementation doesn't have that feature yet.

@Spikeysanju I would like to work on this.

@yogeshpaliyal Thanks for your interest. You can pickup this task 😃 👍
💡 Solution I've considered
Currently we have this Due Date field in our model class. But we're not collecting Due Date & Time from the frontend.
The goal of this task is to add another datepicker field on Add Task & Edit Task Screen.
After that we need to show notification to user based on the due date.
🎯 Action Items

Add Date & Time Picker Field
Show notification to user

P.S - If you have any alternative solution kindly please let me know. Thanks in advance.

@Spikeysanju
What will be trigger for notifications.
Like 1 day earlier, 1 hr. Earlier.

@yogeshpaliyal IMO 1hr Earlier would work. Any thought from your side?

I think we can add 2 notifications first one is 1 hour prior and second 5 minutes prior.

Sounds good to me. Please proceed 🙌

@yogeshpaliyal  PR merged 🚀 . This is huge. Thanks for your valuable contribution 🥇 . Have a great day!

Thanks @Spikeysanju


Make install fails on mac os for PHP 5.6.40

Last 15 lines from<EMAIL_ADDRESS>__php_iconv_mime_decode in iconv.o
_php_iconv_stream_filter_dtor in iconv.o
...
"_iconv_open", referenced from:
_zif_iconv_substr in iconv.o
_zif_iconv_mime_encode in iconv.o
_php_iconv_string in iconv.o
__php_iconv_strlen in iconv.o
__php_iconv_strpos in iconv.o
__php_iconv_mime_decode in iconv.o
_php_iconv_stream_filter_factory_create in iconv.o
...
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [sapi/phpdbg/phpdbg] Error 1
If reporting this issue please do so at (not Homebrew/brew or Homebrew/core):
https://github.com/exolnet/homebrew-deprecated/issues

Hi!
Sorry for the late reply.
Even thought you submitted this issue first, this seems to be a duplicate of #9, but I cannot tell without more context.
Regards.


Wordpress wpskeleton ddev composer create deprecation warning about package name, composer.json

Describe the bug
The first relevant line of a composer.json created by ddev composer create looks like this:
"name": "vendor/.tmp_ddev_composer_create_fldfwf"

Composer complains quite a lot about this, saying that that 'package name' will be invalid in 2.0.

Deprecation warning: Your package name vendor/.tmp_ddev_composer_create_fldfwf is invalid, it should have a vendor name, a forward slash, and a package name. The vendor and package name can be words separated by -, . or _. The complete name should match "a-z0-9/a-z0-9". Make sure you fix this as Composer 2.0 will error.

But the reality is that "name" is naturally the 2nd part of the package name. For example, if the package is "wordpress/skeleton", the "name" will be "skeleton"
My bet is that we need to composer create-project into its own directory named for the top-level package (the natural way)
To Reproduce
ddev composer create wordpress/skeleton --no-install
ddev composer install

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Version and configuration information (please complete the following information):

Host computer OS and Version: [e.g. Windows 10, macOS High Sierra]
Docker version information (use docker version) and insert it here.
ddev version information (use ddev version)
config.yaml contents for the misbehaving project
Do you have any custom configuration (nginx, php, mysql) in the .ddev folder? If so, have you tried without them?

Additional context
Add any other context about the problem here. Thanks!

The weird thing is it shouldn't be creating anything in vendor folder. I'm looking forward to spending some time with wpskeleton...

https://github.com/drud/ddev/issues/1928 points out that wordpress/skeleton isn't really maintained and doesn't work right at all anyway, so not going to spend effort on this one.


Fix: Pagination to consider limit value while returning response.

Pagination fix
Problem?
Often time in production we don't want to return all matching records instead returning a paginated response based on values of limit and offset is desirable.
What is being fixed?
Even after setting a limit value it was not working and .all() or .execute() function were returning complete set of records matching the search criterion.
How is it fixed?
To break away from loop executing the query a condition is added wherein if total number of fetched records is equal or greater than limit.
NOTE: If no limit is provided, default value i.e. 10 records will be returned. Returning all records is usually a bad idea. Take Elasticsearch or Apache solr for example where if you don't provide a limit, by default 10 records are returned.

Any input will be appreciated


Error with calculating log-probs for mask tokens

In huggingface.py the _prepare_text class method takes as input token_ids, and ought to return a list of token_ids with masks in each effective position. However, the code throws an error if the tokenizer that is used is setup to not add bos tokens. Line 694 excludes the first and last token. However, the first token might need to be masked too if it is an effective token. Moreover, our tokenizer has an eos_token only, but line 687 doesn't check for that. I think that line should be changed to:
 effective_token_ids = [token for token in token_ids if token != self.tokenizer.pad_token_id and token != self.tokenizer.cls_token_id and token != self.tokenizer.sep_token_id and token != self.tokenizer.bos_token_id and token != self.tokenizer.eos_token_id]

to be applicable to all possible models.

I also need to comment line 694 and change line 691 to the following:
mask_indices = [[mask_pos] for mask_pos in range(effective_length)]

Originally, it had effective_length+2. No idea why you need to add 2 then immediately remove it?

Another comment. The offset needs to be set to zero as well. So line 701 needs to change to:
masked_tensors.append((torch.stack(token_ids_masked_list), torch.stack(attention_masked_list), effective_token_ids, len(mask_indices), 0))

This issue still persists and the blimp eval code doesn't run for my encoder type model in the latest commit.


subject state attribution - 2 new suggestions

Hi all, I would like to suggest two new attributes in subjectState:

congenitally blind
acquired blind

For now I have added 'congenitally blind individuals' under additional remarks in the subjectState. Or do you think this solutions is appropriate?

@lzehl I agree. They are better suited in the Disease terminology

Yes, I would argue that both are pathologies of the individuals and "congenitally blind" certainly falls under the "disease" category in openMINDS.
I would add "congenital blindness" as a controlledTerms/disease with the following definition:
Congenitally blind individuals were born with permanent and irreversible lack of or severely impaired vision.
The description could elaborate:
Congenital blindness refers to a group of diseases and conditions occurring in childhood or early adolescence. It can be caused by an underlying disease (e.g., optic nerve hypoplasia) or complications during pregnancy or birth (e.g., hypoxia, an infection during pregnancy, premature birth), and typically results in severe visual impairments or blindness. If left untreated, congenitally blind individuals are likely to be permanently blind later in life.
As far as I can see  "acquired blind" refers to individuals have lost their vision later in their life but could see as e.g. children/adolescent. This would be my suggestion for the controlledTerms/disease instance:
"acquired blindness"
definition: A condition in which individuals had little to not impairments in their eye sight (and development of their vision) but have become blind at a later stage in their life (e.g., during adulthood).
description: Acquired blindness refers to a group of diseases and conditions typically occurring in adulthood where an individual had normal eye sight but lost it later on. Causes are widespread. Individuals could have lost their eye sight because of head trauma, other accidents or diseases (e.g. retinitis pigmentosa, (childhood) meningitis, glaucoma).
@AnnaHilverling, would this work for you?

Hi all, thanks so much for all your input.
@UlrikeS91 Thanks for working out the descriptions and definitions.
But I don't know exactly if I can agree that congenital blindness or acquired blindness is entered as a disease. I would rather say that it is a resulting condition from a disease and should not be defined as a disease itself.
What do you think?
As already stated in the definitions above it is a "condition", and as stated in the descriptions above these conditions "refer to a group of diseases" (from which this condition arises as a result).

Following the definitions I found online, I would argue blindness might be best described as a disorder.
A disease is a pathophysiological response to internal or external factors.
A disorder is a disruption to regular bodily structure and function.
A syndrome is a collection of signs and symptoms associated with a specific health-related cause.
A condition is an abnormal state of health that interferes with normal or regular feelings of wellbeing.
The question is whether we are categorising all the above in one single category "disease" or that we want to give some nuance to them.

I understand the confusion, and semantically agree with both @AnnaHilverling and @MaaikevS that blindness is not a disease. Our original idea was to basically collect every abnormal pathology/deviation from normal pathology in "disease".
@AnnaHilverling ignore the name "disease" now - Would you agree that blindness is a pathological change in the body (e.g., definition of "pathologcial": altered or caused by disease)?
If yes, then you would have to register it under "pathology" for the specimen (choosing from controlledTerms/disease or controlledTerms/diseaseModel).

Hi @MaaikevS and @UlrikeS91, many thanks for your reponse.
@UlrikeS91 I understand your original idea. So, yes, I agree that blindness is a pathological change in the body caused by a disease.


chore(RHTAPWATCH-566): Write service monitor to scrape the metrics

Add a Servicemonitor file
metrics should be available on the /metrics endpoint
Add print statement for dsexporter.go when the server starts

Closing this PR as I lost access to my current github account. Created a new one
https://github.com/redhat-appstudio/o11y/pull/153

Does it actually work? Are you able to query in Openshift for the metric the exporter is generating?


Branch UI in status bar doesn't update when switching projects

Version: 1.48.0-insider
Commit: e7920dce7bfd0b707ebfc0a5379c6edd2233e475
Date: 2020-07-10T11:55:11.286Z
Electron: 8.3.3
Chrome: 80.0.3987.165
Node.js: 12.13.0
V8: 8.0.426.27-electron.0
OS: Darwin x64 18.7.0
Steps to Reproduce:
I had a project open where I had made a new branch ("sana-ui-updates" in the image). I opened a new window and new folder with a project that had not been initialized yet. The branch UI in the status bar still shows the branch from my previous project.

Hmm I can't reproduce this. What does the SCM view show?


Do not set parallelism for StreamExecutionEnvironment

What problem does this PR solve?
for flink sql mode,if we set the parallelism for StreamExecutionEnvironment ,  and when user set the parallelism on web ui,
it will be add to flink configuration, so it  will do not effective，so I think we should remove the setting for StreamExecutionEnvironment .

Thank you very much for your contribution. A better way for this bug is to set the parameters in KubernetesNativeSubmitTrait.


Shape/Border overdrawn by Widget

The Shape or Border is overdrawn by other Widgets like the Container or AppBar Widget.
This behaviour exists since  v0.7.1, in v0.7.0 this wasn't happening.
In the example the material widget is overdrawn by the appbar. In reality it should have rounded Corners at the top. One use case where my material widget had rounded corners was when showing rounded bottom sheets.
Material(
      color: Colors.red,
      borderRadius: BorderRadius.only(
          topLeft: Radius.circular(24.0), topRight: Radius.circular(24.0)),
      child: Column(
        children: <Widget>[
          AppBar(
            title: Text("Title"),
          ),
          ListTile(
            title: Text("Item1"),
          ),
        ],
      ),
    );

Please add the output of flutter doctor -v.
A screenshot would be helpful as well.

Sure, that's not a problem.
Here is the output of flutter doctor -v

[√] Flutter (Channel dev, v0.7.5, on Microsoft Windows [Version 10.0.17134.228], locale de-DE)
• Flutter version 0.7.5 at C:\Development\flutter
• Framework revision eab5cd9853 (2 days ago), 2018-08-30 14:47:04 -0700
• Engine revision dc7b5eb89d
• Dart version 2.1.0-dev.3.0.flutter-760a9690c2

[√] Android toolchain - develop for Android devices (Android SDK 28.0.2)
• Android SDK at G:\Porgrammdaten\Android_Sdk
• Android NDK location not configured (optional; useful for native profiling support)
• Platform android-28, build-tools 28.0.2
• ANDROID_HOME = G:\Porgrammdaten\Android_Sdk
• Java binary at: C:\Program Files\Android\Android Studio\jre\bin\java
• Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1024-b02)
• All Android licenses accepted.

[√] Android Studio (version 3.1)
• Android Studio at C:\Program Files\Android\Android Studio
• Flutter plugin version 27.1.1
• Dart plugin version 173.4700
• Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1024-b02)

[√] Connected devices (1 available)
• Android SDK built for x86 • emulator-5554 • android-x86 • Android 9 (API 28) (emulator)

• No issues found!

So here is the example with the bottomsheet, the corners are rounded using this code:
showModalBottomSheet(
        context: context,
        builder: (BuildContext context) {
          return Container(
            height: 250.0,
            color: Colors.red,
            child: Material(
              shape: RoundedRectangleBorder(
                borderRadius: BorderRadius.all(Radius.circular(24.0)),
              ),
              child: SingleChildScrollView(
                child: Column(
                  children: <Widget>[
                    AppBar(
                      title: Text("Header"),
                    ), //Removing this shows the difference
                    ListTile(
                        title: Text(
                      "Item1",
                    )),
                    ListTile(
                        title: Text(
                      "Item2",
                    )),
                    ListTile(
                        title: Text(
                      "Item3",
                    )),
                    ListTile(
                        title: Text(
                      "Item4",
                    )),
                  ],
                ),
              ),
              color: Colors.white,
            ),
          );
        });

And the two screenshots, one with the AppBar(which shows the issue) and one without. In pratice the background is transparent, but i wanted to make it as clear as possible. In v0.7.0 and earlier the rounded corner is not overdrawn.

Material not longer clips (to its shape) by default.  You'll need to specify clipBehavior: Clip.antiaAlias.
cc @liyuqian

Yes, please see https://groups.google.com/forum/#!searchin/flutter-dev/liyuqian@google.com|sort:date/flutter-dev/XMHE0XdsXxI/nonrNQpbBAAJ for why we disabled clip by default.

That solves everything, thank you for the useful information.
Now everything works as expected.

No problem. Thank you for using Flutter!


Docker setup

Hi, I'm working on my final bachelor thesis and I found it really interesting this proposal on a Data Visualization tool.
Since I'm also working on a Data Visualization project, I'm willing to try your app and get some practice on the area.
Have you heard about Docker? I normally use it in order to run applications locally and I would like to know if you are interested in getting it setup in your application. I could work on it and send a pull request.
Let me know what you think.
Thanks and congrats for the project.

Sure let's talk!
Currently im working in a startup so if you are interested in future works / research on CivisAnalysis idea look for Rodrigo Moni<EMAIL_ADDRESS><EMAIL_ADDRESS>he is a master's student at UFRGS and have some related work going on.
The app is running at http://www.inf.ufrgs.br/~fgborja/civisanalysis2/
(Old version with correlation graphs -> http://www.inf.ufrgs.br/~fgborja/civis3/)
You can run the app locally:..

In the /public folder run in terminal 'python -m SimpleHTTPServer'.
-- OR --
install the npm modules 'npm install'
run the grunt 'grunt serve'
The grunt will open http://localhost:9000/ with auto-relaod when editing js/css and compiling the .jade files.

Sorry for the unstructured codes :/ it was my first web/js experience
Have fun!

Thanks for the interest!
Well, I have a pull request ready with the Dockerfile settings and some info about how to run the project with or without Docker. I will add the info you provided and submit it today.
I am very interested in helping with some issues you may have for this project or for some other related work. I'll contact Rodrigo Moni so we can see what I could help you guys with.
Thank you for all the information provided.


Issues and pull request  templates (0/0.5)

Hello,
Currently, issues and PR have different levels of detail in descriptions. It would be useful to introduce templates, which will increase the quality of the descriptions.
https://help.github.com/en/articles/about-issue-and-pull-request-templates
Thanks

I noticed that "Community" section in Github had ready-to-use templates for bugs and new feature requests. I adapted it a bit and created a #191 issue and corresponding PR


Fix the hierarchy of the config file

The connection level settings were not being given precedence over the makara configuration when getting instantiated. This corrects that behavior by properly prioritizing the merging of the configuration options.
This was uncovered while trying to set the blacklist_duration differently on master than on the slaves.


Alexey-Ben meeting 2018-12-20

AGU recap

Mike said he got a bunch of questions about trait covariances in PEcAn's meta-analysis. This will be my next priority for PEcAn development, after we make some significant progress on FoRTE.
Similarly, this means it would be good to get my multivariate meta-analysis paper submitted (and hopefully published!) soon.
ED RTM project

Mike and Shawn were pleased with the results I presented -- they were an improvement over the results from my dissertation. Mike felt this work was approaching publication readiness.
Tossed around a few ideas about ED RTM with Tristan Quaife (U. Reading; works on JULES model; expert on radiative transfer, especially in ecosystem models). He has a new two-stream model implementation that could potentially be an alternative to ED's current one. He also has a snow albedo model, a linearized BRDF model (possibly necessary to convert ED RTM's exitance/hemispherical reflectance output to directional reflectance, which is what remote sensing actually observes), and an alternative implementation of a two-stream model with an analytical solution that could be interesting to compare against ED's.
Mike and Shawn also really want to demonstrate at least a single time-step of state data assimilation using the ED RTM approach (which was the final deliverable of their NASA TE grant that paid my grad school salary before I got my NASA ESSF). My main responsibility here is making sure the ED soil-leaf-canopy RTM works properly (i.e. is not systematically biased) -- some combination of Istem Fer and/or Ann Raiho (and/or others) would take the lead on developing and performing the assimilation step, though I might provide guideance or other help.
Mike, Shawn, and I (and probably Istem, Ann, and Mike and Shawn's current postdocs) will probably have a call about this sometime in January/February to discuss this.

Had a long conversation with Shawn Serbin and Jin Wu (his current postdoc; about to start a faculty job somewhere in China) about possible future collaborations. Most promising idea was to use our Bayesian canopy RTM inversion methods (with some tweaks) to generate global map of leaf/canopy traits and their uncertainties. This would definitely require writing a proposal, probably justified by a small-scale proof-of-concept analysis. My thinking was that the proof-of-concept (map traits for a particular AVIRIS flight, a particular Landsat path-row, etc.) could be a good LDRD proposal at some point.
Another ED2 community meeting will probably happen sometime soon, probably in Boston (hosted by Paul Moorcroft's lab at Harvard). I think this would be a useful meeting for me to attend. I asked his PhD student Miriam Johnston (doing something very similar to me with ED RTM, but for thermal) to keep me posted.

Goals for this week

FoRTE

Start a defensible ED2 100-year ensemble control run at UMBS

Set up PFTs for UMBS
Create a data-driven distribution for PFT traits/parameters (based on TRY data, my RTM-based calibration, multivariate meta-analysis)

May require a bit of hacking around PEcAn's existing code so I can draw parameters jointly (i.e. account for their covariance)

FoRTE first paper outline

Questions and objectives
Methods

Hector

Background reading
Create and try running one permafrost emissions scenario (as proof-of-concept, and just to make sure I understand how to do it)

Other

~Peer review for Geophysical Research Letters (requested a few weeks ago, but I asked for an extension through this Friday because of AGU)~ Update: Completed on Monday, 12/17
(Time-permitting) Submit multivariate trait analysis.

Other notes

First-authored papers in the pipeline

Multivariate TRY analysis (dissertation chapter 1; in prep for Ecology Letters)

Already published as pre-print. Close to submission ready (estimate 0.5-1 days work) for Ecology Letters.

Already formatted for Ecology Letters. Only changes I need to make are (1) flip the order of subsections in the results (and tweak text to make sure it still makes sense) and (2) flesh out the concluding paragraph. Mike had a few other ideas about things to add to the results/discussion, but this never came up in previous reviews so, in the interest of time, I'm inclined to ignore them for now and revisit if requested by reviewers.

Relevant to FoRTE (2) because trait covariance can provide additional parameter constraint. Can cite the preprint for initial submission.

ED2 parameter and structural uncertainty analysis at UMBS (FoRTE year 1 paper)

Outline stage
Aim to submit by end of May 2019?

Hector permafrost emissions paper

Pre-outline stage
Aim to submit by end of May 2019?

ED RTM calibration (AGU presentation; dissertation chapter 4)

Complete rough draft done (dissertation chapter) but needs substantial revision, and to be updated with latest results. Analyses need to be cleaned up. Estimate 3-4 weeks of work.
Relevant to FoRTE (2) because it can help constrain canopy strucutre parameters (which generally lack good trait data) for ED. Can always cite as in prep for initial submission.
Aim to submit by end of May 2019?

Global(ish) leaf spectra analysis using PROSPECT (dissertation chapter 3)

Decent draft in dissertation. Needs to be reframed (per committee comments), but content is fairly solid. Estimate 2-3 weeks of work.
Relevant to FoRTE (2) because it provides an informative prior for leaf optical properties. Can cite as in prep for initial submission.

FoRTE year 1 experimental response paper (and prediction)

Idea stage
Goal: Generate (pre-register?) an ED2 prediction of the experimental responses before we start observing them

Reach goal: Generate ED2 predictions for different killing scenarios before we do the experiment (to inform which trees we kill)

Submit by end of August 2019?

This paper is "Here are our predictions for the FoRTE project." Then, follow up with papers as updates about the status of our predictions for subsequent years. Publication of model forecasts is the ultimate form of pre-registration.

Impact of multivariate constraint on model uncertainty projections

Pre-idea stage
Paper to accompany implementation of multivariate meta-analysis in PEcAn.

FoRTE paper idea -- Maybe re-frame / exend the scope to include drivers of uncertainty in ED2 predictions:

Parameteric uncertainty (by running ensembles over trait uncertainty)

Consider the role of trait covariance? E.g. Here are predictions with vs. without accounting for covariance. But, this might be better off as a separate paper when I fully integrate the multivariate meta-analysis into PEcAn

Structural uncertainty (our factorial ED "scheme" experiment)

Besides the processes we are already thinking about, an interesting interaction between structural and parameteric uncertainty is PFT definition. E.g. We could run one ED2 ensemble with a standard PFT specification ((early, mid, late) x (hardwood, conifer)), and another with more finely resolved PFTs, maybe even down to the species level (e.g. aspen, birch, red oak, red maple, pine). This would increase the parameteric uncertainty somewhat, and could expose problems related to limited similarity. The latter opens up an interesting discussion about real, significant differences between species that are not represented by ED (e.g. clonal reproduction of aspen but not birch, different ectomycorrhizal associations)

Driver uncertainty (i.e. meteorology data)?

I saw several AGU talks (including one from Gordon Bonan) that highlighted how different various meteorological reanalysis products are and how that can affect predictions about various processes (Bonan showed it's an important source of uncertainty for CLM projections)
One challenge is finding climate data products that go back to 1900 that have all the variables ED needs (in my experience, downwelling radiation is a variable that is often missing) and have a good-enough temporal resolution (at least 6-hourly). If there's at least one other one that works more-or-less off the shelf, this shouldn't be hard to do. An alternative is to try to use some stats to derive some of the other variables and/or downscale, but that's complex-enough to warrant its own paper.
Another reason to punt on this for the first paper is that adding another factor to our factorial experiment represents a significant additional computational burden.

Also, just spoke to Mina and it sounds like we should briefly discuss my billing targets for the next year? She mentioned that you may want me to reduce FoRTE and add some LDRD-related work...?

Useful, and interesting–thanks. A few other notes, esp. wrt papers:

Re driver data uncertainty, yes, I've seen this a bunch too. As part of E3SM work, I have site- and global-scale runs using two different BGC modules and two different driving datasets (GSWP3 and HadCRUT), i.e. a structural versus driver data uncertainty test. I'm supposed to lead a paper on this in 2019, and your expertise would be useful and welcome.
"Signal emerging from noise" paper–what you and Jinshi and I talked about a bit, based on my Friday talk. There are a number of different ways this could go, some more directly relevant to your areas of expertise and some less, but regardless it would be great to have your involvement if of interest.
FoRTE paper on structural/parametric uncertainty–seems promising.
Glad Mike and Shawn were pleased with development of ED RTM work! 👏 Aiming to submit this within 4-6 months seems good.

See you tomorrow, B


Create User -> Create Certificate missing 'Digest Algorithm' options

http://i.imgur.com/IQJtgI1.png
Also option to add a few alt names would be nice.

Creating a client certificate for a user (email as CN) using cert manager
http://i.imgur.com/5hRXnFq.png
Creating a user with selecting the option to create a certificate
http://i.imgur.com/QlKyCIC.png
As you can see 'Digest Algorithm' option is missing. (I'm also not sure if the certificates are generated properly with the 'Client Authentication' DN, but that's something to check out at a later time. I'll file a new report if that is not the case.

I have just re-factored all the cert manager pages, I'm going to solve the inconsistency issue by redirecting to the cert manager from the user manager instead of duplicating code there....


[5.9] backport platform modularisation changes for Windows

This cherry-picks a number of changes that have been applied to the platform modularisation on main to enable C++ interop (and some dependencies).  This enables fixes to the use of std.map on Windows.
This is a low risk change as it impacts Windows only.  Any issues would be caught implicitly by the build on the release branch.

@swift-ci please test


Refactor BWC test structure and build.gradle

Description
This change refactors the BWC test suite as well as the build.gradle file. I moved the BWC code into a directory inside source and added the tasks associated with it to the main build.gradle file. I also went through and tried to simplify the code in the build file as much as possible.
Issues Resolved
https://github.com/opensearch-project/security/issues/3881
Is this a backport? If so, please add backport PR # and/or commits #
Testing
Ideally all tests that used to run should still run.
Check List

[ ] ~New functionality includes testing~
[ ] ~New functionality has been documented~
[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Closing this for the time being as I don't have the time to finish it right now.


KeyError: 'ref_junction_map'

I get the following error when I try to train on wireframe dataset.
Wireframe doesn't have this key.
python -m sold2.experiment --mode train --dataset_config sold2/config/wireframe_dataset.yaml --model_config sold2/config/train_full_pipeline.yaml --exp_name sold2_full_v0.5.7 --pretrained --pretrained_path pretrained_model/ --checkpoint_name sold2_wireframe.tar
    junc_map = data["ref_junction_map"].cuda()
KeyError: 'ref_junction_map'

@rpautrat I am sorry but there are so many terminologies in SOLDv2 work which is making to complex to adapt.
what is different in "official" and "exported" dataset ?
Can I directly jump to step5 for training on my custom data ?

I still get the same error when I run step 5 directly with gt_source_train="official" and gt_source_test="official"

When I try step 4 I get the following error -
  File "/home/ubuntu/abhishek/SOLD2/sold2/train.py", line 254, in train_single_epoch
    heatmap = data["heatmap"].cuda()
KeyError: 'heatmap'

I used the command -
python -m sold2.experiment --mode train --dataset_config sold2/config/wireframe_dataset.yaml --model_config sold2/config/train_detector.yaml --exp_name sold2_wireframe --pretrained --pretrained_path pretrained_model/ --checkpoint_name sold2_synthetic.tar

Now that I remember, the code is actually not made to train on the official ground truth. We only used it to evaluate the line descriptors on these lines, but it is currently not compatible with training.
What you could do for step 4 would be to modify the return function of the wireframe dataloader https://github.com/cvg/SOLD2/blob/3d7bcd602ff4fb44e118d745a10251c829fa0ab1/sold2/dataset/wireframe_dataset.py#L548-L559
If you modify the keys 'line_map_pos' to 'line_map' and 'heatmap_pos' to 'heatmap', it might work.
Step 5 with the 'paired_desc' option is not available at all currently. It would need additional changes to the dataloader.
But again, the goal of SOLD2 is to train without existing ground truth, so it doesn't make sense to use the official ground truth, which is not good for localization for example.

Regarding the error "TypeError: warp_perspective() got an unexpected keyword argument 'flags'", the issue is with your kornia version. We used an older one that had different arguments. Please consider using kornia==0.3.0 and it should solve it.

Regarding the error "TypeError: warp_perspective() got an unexpected keyword argument 'flags'", the issue is with your kornia version. We used an older one that had different arguments. Please consider using kornia==0.3.0 and it should solve it.

I updated kornia==0.3.0  and Now it installed torch==1.5.0. Now its giving me error -
File "/home/ubuntu/anaconda3/envs/sold/lib/python3.8/site-packages/torchvision/ops/boxes.py", line 78, in <module>
    @torch.jit._script_if_tracing
AttributeError: module 'torch.jit' has no attribute '_script_if_tracing'

@rpautrat Where do I give path to my custom data ?


New feature: enable validation webhooks

Scaffolding using the operator-sdk
Adapted the code just enabling Validation webhooks for Create and Update verbs.
Implement custom logic that validates the module's specs.

/cc @ybettan @yevgeny-shnaidman @mresvanis

/lgtm


GUI lacks "log out" button

The OSX client has no way to log out or switch accounts, either from the main app or from the status bar interface.
"keybase logout" from the command line works.

Yeah... I just installed this on Windows, where I'm not familiar with the CLI, and signed in to the wrong account. I can't figured out how to sign out.
I'll probably end-up deauthenticating the device from another device and hoping that has the right effect, but not having a clear way to log out/deauthenticate on-device is a pretty major omission.

@jeremyBanks Ah, the "keybase logout" command line mentioned works from the "Keybase Shell" Windows start menu item.

I don't think this is the case anymore. I see a Sign out button on the settings page of the Mac app - OS X client. Hopefully that is what you were referring to as well, @longears?

This response is now the right answer, so this issue can be closed, @keybase. IMO.