[Bugfix] Move brokerId extraction to BaseBrokerStarter

BrokerRequestHandlerDelegate is the first place where broker request ever gets compiled. If there is any exception on compiling the request, the requestContext will miss some of the information like brokerId.
Plus, there are a lot of places where this brokerId gets extracted, there should be only 1 place for doing that.
This PR moves brokerId extraction up to BaseBrokerStarter.

Codecov Report

Merging #9965 (97465e3) into master (b3d1ac3) will decrease coverage by 45.26%.
The diff coverage is 69.23%.

@@              Coverage Diff              @@
##             master    #9965       +/-   ##
=============================================
- Coverage     70.37%   25.11%   -45.27%
+ Complexity     5071       44     -5027
=============================================
  Files          1987     1975       -12
  Lines        106761   106415      -346
  Branches      16203    16168       -35
=============================================
- Hits          75134    26725    -48409
- Misses        26385    76977    +50592
+ Partials       5242     2713     -2529

Flag
Coverage Δ

integration1
25.11% <69.23%> (-0.06%)
:arrow_down:

integration2
?

unittests1
?

unittests2
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...roker/requesthandler/GrpcBrokerRequestHandler.java
0.00% <0.00%> (-86.96%)
:arrow_down:

...e/pinot/broker/broker/helix/BaseBrokerStarter.java
73.54% <50.00%> (-5.82%)
:arrow_down:

...roker/requesthandler/BaseBrokerRequestHandler.java
59.27% <100.00%> (-13.36%)
:arrow_down:

...r/requesthandler/BrokerRequestHandlerDelegate.java
60.60% <100.00%> (+2.54%)
:arrow_up:

...requesthandler/MultiStageBrokerRequestHandler.java
58.90% <100.00%> (-0.56%)
:arrow_down:

...thandler/SingleConnectionBrokerRequestHandler.java
88.31% <100.00%> (ø)

...in/java/org/apache/pinot/spi/utils/BytesUtils.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...java/org/apache/pinot/spi/trace/BaseRecording.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...java/org/apache/pinot/spi/trace/NoOpRecording.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ava/org/apache/pinot/spi/config/table/FSTType.java
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 1446 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Replying to a message from the web doesn't actually reply

One time it gave a 415, unsupported medial type error. ("text/plain;charset=UTF-8").
Usually it just looks like a normal reply

This is probably out of scope for the moment, but I might look at it quickly later


No interaction with world

We are using this docker on an Ubuntu 20 system. We are connecting via local Wifi only. Everything works perfectly until after 15-30 min player (3) start to "de-sync" with the server. One can run around but not interact properly with the environment. Reconnecting is the only way to solve it.

Sounds more like a server than a container question. But maybe with some more info somebody can help.
Like what's the hardware the server is running on? Valheim likes lots of RAM (it'll happily take 10 GB although 4 is enough if it is allowed to swap) and two or more high clocked cores. Performance is also dependent on the slowest player for whatever reason. Like we host our server in EU and whenever a friend from Australia joins in everyone starts to lag. Though not to the extend you describe. But since you're saying you're on Wifi and I don't know the quality of your Wifi (are you living alone in the countryside or in an apartment complex with 30 other SSIDs competing for airtime) it's possible this could be an issue.
There is no general "start lagging after 15-30 minutes" that affects everyone if that was your question. Normally you should be able to play for 10h straight with no performance issues.

Server hardware has 8GB RAM with 2GB swap. I monitored the memory consumption and it is never above 30%.
You have a point regarding the WiFi SSIDs. We live in the city centre. The server is connected via cable to the local network but I might check if this happens if the host uses LAN, too.
By now, I don't think it's a container problem as well. The container works great, thank you. Yesterday, we played 1,5 hours with no problems at all. And then de-sync after every 15 min and then it might happen that the problem persists after host reconnection. When this happens, I have to restart the server.

We definitely noticed the server leaking resources after a while. I don't know if that's still the case but a couple of weeks ago when I first build the container it happened for sure and a restart would make everything smooth again. Though for us it happened only after like 2-3 days of running continuously. That's how the 5 AM RESTART_CRON was born. We just restarted the server once a day and the issues were gone for us.

@andrejberg were you able to figure out what caused the lag and desync? Is it still an issue?

Yes, it was a network issue. We have now connected all machines via LAN and the problem is solved completely. I guess it is the crapy 8-year-old router that is not able to provide a stable WiFi signal. Thank you @lloesche and @Addyvan for your help.


fix: Fix date/datetime_to_string handling of non-date/datetime inputs

This PR fixes the behavior of the recently added date/datetime_to_string methods to gracefully handle inputs that are not date or datetime.  Most importantly, these methods should return None if the input is None.  The new implementation simply returns the input argument whenever it is not of the expected type, which is consistent with the behavior of the other methods in the utils file.

:tada: This PR is included in version 1.6.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


adminserver安装board时启动不了，和gitlab有关

In GitLab by @cuiaq123 on Jul 29, 2020, 09:16

In GitLab by @wknet123 on Jul 30, 2020, 01:49
Gitlab should not be installed with Board on the same VM/server when it has memory less than 8G because the resource of VM/Server is too limit to run both of two large memory costing Docker containers.

In GitLab by @wknet123 on Jul 30, 2020, 01:49
assigned to @cuiaq123 and unassigned @wknet123

In GitLab by @cuiaq123 on Aug 10, 2020, 09:35
assigned to @wknet123 and unassigned @cuiaq123

In GitLab by @luotian-github on Sep 8, 2020, 24:58
assigned to @cuiaq123 and unassigned @wknet123

In GitLab by @luotian-github on Sep 8, 2020, 24:58
assigned to @guyingyan and unassigned @cuiaq123

In GitLab by @luotian-github on Sep 8, 2020, 24:58
add some checks for users to install gitlab


etcd-01 Unhealthy   Get https://<IP_ADDRESS>:2379/health: remote error: tls: bad certificate

I followed this tutorial, but I get an error
etcd-1               Unhealthy   Get https://<IP_ADDRESS>:2379/health: remote error: tls: bad certificate
And all the other etcd nodes when trying to do
kubectl get componentstatuses
in step 5. What this can be related to?

I hit the same problem and concluded that it might have something to do with https://github.com/kubernetes/kubernetes/issues/29330 which should be resolved by https://github.com/kubernetes/kubernetes/pull/39716
But this is only a wild guess.

Testing with the following reports that it's healthy, so it does indeed seem to be a problem with Kubernetes:
sudo etcdctl --debug --endpoints https://<IP_ADDRESS>:2379 --ca-file /var/lib/kubernetes/ca.pem --key-file /var/lib/kubernetes/kubernetes-key.pem --cert-file /var/lib/kubernetes/kubernetes.pem cluster-health

Seems that this is just an error with the report on health (which the issues that @temal are also claiming).  I finished the remaining steps, and everything worked.

Using kubernetes ´v1.7.0-alpha.3´ it works very well the PR https://github.com/kubernetes/kubernetes/pull/39716 is merged in https://github.com/kubernetes/kubernetes/releases/tag/v1.7.0-alpha.3

Just a note to say that I am getting the same, but I haven't tried with a v1.7 version yet...

yeah, I got the same error here. After upgrading to latest "v1.7.0-alpha.3", problem solved.

Same here, upgrading to latest "v1.7.0-alpha.3" solves the issue.

@Bo0mer - Could you share the steps to upgrade to "v1.7.0-alpha.3"? I am not finding the actual binary.

Just replace v1.6.1 in the urls with v1.7.0-alpha.3, e.g. https://storage.googleapis.com/kubernetes-release/release/v1.7.0-alpha.3/bin/linux/amd64/kube-apiserver.
And there is alpha.4 already, so I guess you could try it also.

Maybe an issue with compatibility? When I check etcd health i get the following:
`controller0:~$ sudo etcdctl \

--ca-file=/etc/etcd/ca.pem
--cert-file=/etc/etcd/kubernetes.pem
--key-file=/etc/etcd/kubernetes-key.pem
cluster-health
2017-06-05 12:18:22.330990 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
2017-06-05 12:18:22.331993 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
member 3a57933972cb5131 is healthy: got healthy result from https://<IP_ADDRESS>:2379
member f98dc20bce6225a0 is healthy: got healthy result from https://<IP_ADDRESS>:2379
member ffed16798470cab5 is healthy: got healthy result from https://<IP_ADDRESS>:2379
cluster is healthy`

My guess is that kubectl is marking the issue as a failure instead of a deprecation warning?

I got the same error - trying the steps twice incase I made a mistake and openssl working fine over TLS: openssl s_client -connect 10.240.10:2379 -CApath ~/

I've updated all the docs. This should be fixed now.


Create differential fuzzers and add them to OSS-Fuzz

This attempts to implement #1003.

Some last things I'm not quite sure what to do on:

[ ] onepass says that it can panic but the try_search API is dissimilar to the others in a way I'm not confident I can do correctly :sweat_smile:
[ ] should we add configuration to the FuzzData? Not all config combos are tested, but maybe this is not important

Found a few disagreements already. Some of them I'm reasonably certain are just misuses of the API on my part. Others appear to be rather rare disagreements caused by unicode word boundary shenanigans again. Copying an example below (I can raise a separate issue but not sure if desired):
#[test]
fn unicode_disagreement() {
    let pattern = ".+\\b\u{a}";
    let haystack = "7\u{ffff}77\u{a}";
    let baseline = PikeVM::new(pattern).unwrap();
    let mut cache = baseline.create_cache();
    let re = Regex::new(pattern).unwrap();

    let found1 = re.find(haystack);
    let found2 = baseline.find(&mut cache, haystack);
    if let Some(found1) = found1 {
        let found2 = found2.expect("Found in target, but not in baseline!");
        assert_eq!(found1.start(), found2.start());
        assert_eq!(found1.end(), found2.end());
    }
}

Let me know how you want me to file these and I'll start working away at it, but I get the feeling that some of these are false positives...

@addisoncrump I think that unit test depends on what Regex is. If it's meta::Regex (or regex::Regex), then yeah, that test should pass for all inputs I believe. I simplified it down a little bit to this:
use regex_automata::{meta::regex, nfa::thompson::pikevm::pikevm};

fn main() {
    env_logger::init();

    let pattern = r".+\b\n";
    let haystack = "β77\n";
    let baseline = pikevm::new(pattern).unwrap();
    let mut cache = baseline.create_cache();
    let re = regex::new(pattern).unwrap();

    let found1 = re.find(haystack);
    let found2 = baseline.find(&mut cache, haystack);
    if let some(found1) = found1 {
        let found2 = found2.expect("found in target, but not in baseline!");
        assert_eq!(found1, found2);
    }
}

Also enabled the logging feature and ran it with RUST_LOG=trace to get an idea of what's actually happening. Indeed, this is a bug, and it looks like an issue in the reverse suffix optimization. Sigh. The return of the dreaded literal optimization bugs. (Although this one appears to intersection with both a Unicode word boundary in the regex and a non-ASCII codepoint in the haystack.)
As for what to do here... Hmmm. I'd say this one is definitely deserving of its own bug ticket: https://github.com/rust-lang/regex/issues/1046
I don't necessarily want you going through the rigamarole of creating a bug ticket for every case you find? I'm not sure. Depends on how many they are and whether they're all the same bug.

There are lots of mismatches on DFA targets, but I think I'm misusing that API. I worry that I won't be able to distinguish between causes or do effective root-cause analysis on these as they are quite in the weeds :sweat_smile: I will do my best to minimise and look into it as much as I can, but if I can't figure it out I'll share a reproducer.

Aye thanks. If you burn out of it (I know going through these test cases is a lot of work) then just sharing the cluster test inputs or whatever would be great. Then I'll be able to go through them (at some point).
And yeah unfortunately the API for regex-automata is incredibly complex. There's a billion knobs. I think my docs should explain everything but, well, there are a lot of docs.

A slight concern I have merging this too early: if we aren't reasonably certain that we haven't gotten rid of at least most of the mismatches before merging, I'm reasonably confident that OSS-Fuzz may fail to report some bugs... I need to look into it further, but I'm fairly certain it deduplicates by stacktrace -- and here, all of our stack traces will be the same for different bugs!
I've asked the OSS-Fuzz folks and will think about how to avoid this problem a bit before merging.

Yes I would probably run this locally (or use your test cases) before merging and try to squash some bugs first. Basically, get it to a point where I can run it over night and not have it find anything. (That's what I did with fuzzers currently on master IIRC.)

Any interest in moving forward with this? We can simply not put them in OSS-Fuzz, as you prefer.

I am definitely interested, but it might take me a while to get back around to this unfortunately.

Okay, no issues. For the sake of making clear what remains: the DFA differential fuzzers seem to encounter issues almost immediately, but I'm not entirely sure if this is my fault. The other fuzzers execute as expected and have already found some bugs.


how to use MessagePack missing

the documentation mention MessagePack protocol but it does not show how to use this protocol, what needs to be installed ? what configuration is needed ? client code ?
keyword msgpack5.js, signalr-protocol-msgpack.js

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 22325a88-5e1d-7b54-87b7-ea331a6c73b6
Version Independent ID: b92a4bf5-f7c9-50b9-d7d7-46ebcf7387b0
Content: ASP.NET Core SignalR
Content Source: aspnetcore/signalr/index.md
Product: asp.net-core
GitHub Login: @rachelappel
Microsoft Alias: rachelap

A doc has been written for this but isn't pushed to the site yet, you can view it at https://github.com/aspnet/Docs/blob/master/aspnetcore/signalr/messagepackhubprotocol.md

thx this document is helpful except it mention you also need to add msgpack5 lib but does not describe the way to install msgpack5.js


supabase-js v2.43.6->v2.44.0 lose auto-inferred type-safety and failing build

Bug report

[x] I confirm this is a bug with Supabase, not with my own application.
[x] I confirm I have searched the Docs, GitHub Discussions, and Discord.

Describe the bug
When using supabase-js v2.43.6 (which bumps postgrest-js to v1.15.6), the supabase client loses auto-inferred type-safety when calling supabase.select("field_name"), and returns an object type of any, which will fail builds that has "no implicit any" rule enabled for TypeScript + ESLint.
To Reproduce
Steps to reproduce the behavior:
Update supabase-js to v2.43.6, which will fails build on https://github.com/tutur3u/platform.
Expected behavior
Build should be passing, as it was passing on the previous version (v2.43.5)
System information

OS: Windows 11 23H2
Version of supabase-js: v2.43.6
Version of Node.js: v22.2.0

Related
https://github.com/supabase/postgrest-js/issues/545
https://github.com/supabase/supabase-js/issues/1229

I am also seeing some types suddenly breaking

I also have those issues, I made a repo to reproduce it:
https://github.com/ex0ns/eslint-supabase
Repro Steps
    clone the repo
    pnpm install
    pnpm exec eslint index.ts

Build fails on v2.44.0 as well.
Ref: https://github.com/tutur3u/platform/pull/1553

Same issue, darwin latest on a nextjs project

Yeah just tested and the issue is on 2.43.5 as well.

try to pin supabase-js to "2.43.5" instead of ^"2.43.5" so that it doesn't unintentionally uses newer version @thisislvca

@vhpx yup alr done that :)

Sorry for the mess! This should be fixed in the latest release.

@soedirgo I'm still seeing build failure on 2.44.2

@ham-evans try 2.44.3? We've found another module-related bug that should be fixed there.


Support "isName()" getters instead of "getName()" for private booleans

Currently for private fields, the guide says to "specify a getName() and setName(columnType) for the field named name."
I'm sure I'm not the only one who uses the convention "isName()" instead of "getName()" in the case of accessors for booleans. IntelliJ/Android Studio uses it by default when using "Create Getter" or "Encapsulate Fields" for example.
I've recently found myself doing quite a bit of refactoring to accommodate DBFlow using getName() for booleans, and it would be really great if DBFlow supported isName() as well.
Thanks, and keep up the great work!

I've run into this as well, would enjoy being able to name things a bit better. is_awesome is different from getAwesome and leads to confusion. I would not want to think just because something is awesome that they could also get it.

will be added in 3.0 as useIsForPrivateBooleans() in @Table

Awesome, thanks!
On Sat, Oct 10, 2015, 7:15 PM Andrew Grosner<EMAIL_ADDRESS>wrote:

will be added in 3.0 as useIsForPrivateBooleans() in @Table
—
Reply to this email directly or view it on GitHub
https://github.com/Raizlabs/DBFlow/issues/407#issuecomment-147133952.

Thanks!

in develop


Hot reload devtools extension

Type:

[ ] bug
[x] feature
[ ] enhancement
[ ] question

Environment:

OS: any
Browser: chrome
Library Version:

I'm going to open a PR:

[ ] yes
[x] no

Description:
I’m currently building a chrome devtools extension (https://developer.chrome.com/extensions/devtools). Unfortunately this plugin doesn’t automatically reload the custom devtools page, although it does reload the background script.
What would it take to extend your extension, in order to hot reload devtools extensions as well?

+1

Not sure, this is specific from Chrome right?
If isn't on the oficial browser extension specification unfortunately can't do anything.
If I add something like this (that I even know if is possible) I'll be dropping all other browsers :/
Sorry about that and thanks for the feedback!


After installing preview-9, unable to run Node.js server

After I installed preview-9, I am getting this error:
/node_modules/@generated/nexus-prisma/index.js:877
        return nexus_1.dynamicOutputProperty({
^
TypeError: nexus_1.dynamicOutputProperty is not a function
    at NexusPrismaBuilder.getCRUDDynamicOutputMethod

Is this a new project created with prisma2 init or an existing one you created yourself before?

@janpio it is an existing project, I just updated prisma.

I can reproduce this.

Reproduction Steps:

prisma2 init
Starter Kit > GraphQL
yarn install
yarn start

I think this belongs to nexus-prisma repository.

I believe getting everything to its latest form should fix this as a related fix was merged here: https://github.com/prisma/nexus/pull/205#issuecomment-524596726


Turn FEI into FAQ

WIP
Turns the Frequently Encountered Issues page into the Frequently Asked Questions page.

[x] Turn all current FEIs into FAQs
[x] Adjust links to FEIs from other pages
[x] Add alias for the FEI page
[x] Add the redirect for the 0.13 FEI to 1.0 FAQ
[x] Add in all TL;DR blog entries (#640)
[x] Add FAQ about how to query a field key/tag key with the same name as a tag key/field key (#682)

Fixes #640, #682.

👍


Add the postgres to HoH operator

Signed-off-by: myan<EMAIL_ADDRESS>Customers give the HoH operator access to the specific database by creating a secret with the following fields
kubectl create secret generic "postgresql-secret" -n "open-cluster-management" \
    --from-literal=host="*" \
    --from-literal=port="5432" \
    --from-literal=user="*" \
    --from-literal=password="*" \
    --from-literal=database="*"

The secret will add two fields[process_uri, transport_uri] when the HoH database is initialized successfully.
» oc get secret postgresql-secret -oyaml
apiVersion: v1
data:
  database: *
  host: *
  password: *
  port: *
  process_uri: *
  transport_uri: *
  user: *
kind: Secret
metadata:
  creationTimestamp: "2022-08-03T09:07:28Z"
  name: postgresql-secret
  namespace: open-cluster-management
  resourceVersion: "6679"
  uid: 4cccf1d7-b066-453f-99d7-b4054217db43
type: Opaque

NOTE: The secret name postgresql-secret can be any value, as long as it is the same as postgresql.name in the HoH CR.

Reference: https://github.com/stolostron/backlog/issues/24506
Fixed: https://github.com/stolostron/backlog/issues/24726

/cc @morvencao

ref: https://github.com/stolostron/backlog/issues/24609


loosing expansion in CHECK due to CAPTURE

If I run
#include <catch.hpp>
#include <vector>

TEST_CASE("capture variable without loosing expansion") {
    std::vector<int> v {1, 2, 3};
    CHECK(2 == v.size());
}

an expansion of the right hand side expression v.size() is printed
/path/to/my_test.cpp:6: Failure:
  CHECK(2 == v.size())
with expansion:
  2 == 3

If I add a CAPTURE
#include <catch.hpp>
#include <vector>

TEST_CASE("capture variable without loosing expansion") {
    std::vector<int> v {1, 2, 3};
    CAPTURE(v);
    CHECK(2 == v.size());
}

I get the expected CAPTURE message, but I loose the expansion
/path/to/my_test.cpp:7: Failure:
  CHECK(2 == v.size())
with message:
  v := { 1, 2, 3 }

I expect the output to be
/path/to/my_test.cpp:7: Failure:
  CHECK(2 == v.size())
with expansion:
  2 == 3
with message:
  v := { 1, 2, 3 }

Is this a bug or a feature?

Catch v1.8.2

What compiler? I took your second example, only added #define CATCH_CONFIG_MAIN on top, and am getting the expected output with both expansion and message. gcc 4.8, 5, 6, clang 3.6, 3.8, 4.0

Are you using a custom reporter? Because I searched for the word "Failure"
git grep -w 'Failure' $(git rev-list --all) -- include/reporters/

and it only appears in the xml reporter.

Ah!
Are you using CLion, by any chance, @maiermic?
(CLion's integrated Catch test runner uses the XML reporter, but then presents it back in a way that looks similar to the console reporter.
That might explain the difference. I'll take a look at that.

@philsquared You are right. I use Clion 2017.1 EAP with a Catch Test configuration. If I use an Application configuration it shows the correct result. Sorry, I should have checked that earlier.

I have updated to the latest version CLion 2017.1 RC2, build 171.3780.103, March 22nd, 2017. Same issue. I created a bug report for Clion.

Perfect - thanks, @maiermic - saves me doing that.

Running your example with the XML reporter confirms that all the information is present in the report - so this is a pure CLion issue.
Therefore do you have any objection to me closing this ticket, @maiermic?


Place Icons before chapter numbers

Done as far as it goes, but as discussed, we may need to create svg icons instead of font icons in the text.

From the OpenXR Tutorial Review Meeting on 2023/08/22 at 14:00 UTC, we are switching to SVG icons before chapter numbers.
I believe Simul will look into finding a graphics designer for these.

First icon is implemented, a hand controller icon for Ch4. Others to follow.

Tricky to get these icons into the sidebar TOC, as these don't have individual html id's. We may need to do this with JavaScript.


Update deep-linking.md - Fix Declare Association between website and …

…intent-filters external documentation url
The current link redirects to https://support.google.com/android/search?q=open+by+default#web-assoc which does not provide the intended document. I changed it to the actual url which is 'https://developer.android.com/training/app-links/verify-android-applinks#web-assoc'
READ ME PLEASE

TL;DR: Make sure to add your changes to versioned docs

Thanks for opening a PR!
The docs cover several versions of react-navigation, and in some cases there are several files (for version 1, version 2 and etc.) that all describe a single page of the docs (eg. "Getting Started").
Please make sure that the edit you're making in docs/file-you-edited.md is also included in the file for the correct version, eg. /versioned_docs/version-3.x/file-you-edited.md for version 3. If such file doesn't exist, please create it. :+1:

@BismarkCodes is your update still true? When I look up instructions on how to add association, Google tells me to use Google Play Store -> App -> Setup -> Advanced Settings -> App Indexing -> Verified Websites. Not pushing back on your PR, just trying to figure out what the right way to do this is... Thanks!


Chunk adding tensor to h5 file to prevent out_of_memory errors

Mentioned in #18

@maxhodak One thing I forgot to mention, this change also makes the preprocessed h5 file ~2X smaller, since this stores the tensor as a "float4" type instead of an "integer8" type, at least on my machine. Don't know if this is an undesired side effect.

The preprocessed file is a huge sparse binary matrix, so it seems like this shouldn't make a difference. There are definitely optimizations to do here but it hasn't been broken enough for me to bother with yet.

I don't think the float vs int encoding should make a difference here but I should probably investigate a little bit.


"docker-compose up export" step fails

Hello,
I have followed the instructions listed at http://osm2vectortiles.org/docs/own-vector-tiles/ with Zurich PBF and it worked (including the visualization).
However, to build my own tiles and for testing purposes, I am using Buenos Aires PBF (https://s3.amazonaws.com/metro-extracts.mapzen.com/buenos-aires_argentina.osm.pbf). Prior to running the export step, I change the BBOX line to BBOX: "-58.89,-34.96,-57.99,-34.29"  as indicated and the export step throws an error in tilelive-copy javascript file (please see attached image).
I have also followed the instructions listed in the thesis and the same error pops up.
I don't quite understand why tilelive-copy is invoked at this point of the process. It is my understanding that tilelive-copy allows to extract a subset of tiles from an mbtiles database. At this point the actual mbtiles database has to be created.
Any help on how to fix this error will be greatly appreciated.
It is also worth noting that this is a great project!
Regards,
Clara

Also had this error once, it is because bbox's starting with a negative number are not handled properly. I found a workaround, just add a space before the negative sign => " -58.89,-34.96,-57.99,-34.29"

We have to investigate, what causes this error.

Also had this error once, it is because bbox's starting with a negative number are not handled properly. I found a workaround, just add a space before the negative sign => " -58.89,-34.96,-57.99,-34.29"

It is exactly that. I think using the named parameter differently https://github.com/osm2vectortiles/osm2vectortiles/commit/01612da95fc31112df8c0183a0b73da373ac961b can fix it (doing something like this in the export Python script).

It is exactly that. I think using the named parameter differently 01612da can fix it (doing something like this in the export Python script).

That was quick 👍

Hi Lukas, Manuel,
Thanks for the quick turnaround!
Now I passed that but even if I have a zoom range of 8 to 14, I get a PostGIS error: "relation "landuse_z5toz8" does not exist"
Any hint on this!?
Thanks!

You have to run docker-compose run import-sql

I've done that already with exit code 0 ...

Oh, now I get it. Can you make a git pull?- This is a very recent change.

doing it right now

Should I just repeat the export step or I would need to start from the beginning?

Just repeating the export step should work.

Ops! Same error:
Error: Postgis Plugin: ERROR:  relation "landuse_z5toz8" does not exist

Would you be up to do a quick skype call?- It is probably easier to troubleshoot. You should find me on skype with this email adresse<EMAIL_ADDRESS>

Did the git pull command fetch any new changes?

Hello Manuel,
It did bring about 33 files changed. It did work now! Awesome! Thanks!

Oh, great to hear.

I have a short question: if I want to add new regions can I just repeat the steps from import-osm on?

I have a short question: if I want to add new regions can I just repeat the steps from import-osm on?

No the process is not intended to add multiple independent regions. The idea is to import a pbf-extract and export as vector tiles. If you want to have multiple regions this is the way to do it.


Monero TODO

[ ] Add libraries

[ ] API for prices

[ ] Payment Box

[ ] Integrated address

[ ] Payment confirmation

[ ] Removing Bitcoin

[ ] Anything else?

I started to work on this, problem is I am good in python but not in php ( I am begginer ). I created Python scripts to handle monero multi-sig, but if anyone is willing to continue this project, I want to help to implement it in annularis.


Fix mock for TextInput

Summary

This PR adds the isFocused method to the mock of the TextInput component. My understanding some of the latest changes on the TextInput to make it use a forwardRef change the way this method is mock giving an error when trying to use in on a mock.
The change suggested here fixes the issue.
Changelog

[JavaScript] [Fixed] - Fix the mock for TextInput to support the isFocused method
Test Plan

Good catch! Can you add the other two methods as well? https://github.com/facebook/react-native/blob/master/Libraries/Components/TextInput/TextInput.js#L694-L698

@TheSavior just update the PR with the changes you requested.


ndarray.dot produces wrong values with non-native endian arrays

Some numpy arrays have non-native endianess. Numpy uses dtype.isnative to denote the endianness of an object and handles both non-native and native endianness gracefully. However, cupy.ndarray.dot seems to ignore this flag. This leads to incorrect values being calculated on the gpu.
Conditions

CuPy Version          : 6.4.0
CUDA Root             : /usr/common/software/cuda/10.1.168
CUDA Build Version    : 10010
CUDA Driver Version   : 10010
CUDA Runtime Version  : 10010
cuDNN Build Version   : 7603
cuDNN Version         : 7603
NCCL Build Version    : 2402
NCCL Runtime Version  : 2402

Code to reproduce
import numpy as np
import cupy as cp

# Create example arrays
A_cpu = np.arange(9.0).reshape(3,3)
x_cpu_native = np.arange(3.0)
x_cpu_nonnative = x_cpu_native.byteswap().newbyteorder() # create a copy of x that has non-native endianness
A_gpu = cp.array(A_cpu)
x_gpu_nonnative = cp.array(x_cpu_nonnative)
x_gpu_native = cp.array(x_cpu_native)

# Simply transferring a non-native array to the gpu and back seems to preserver endianness information
# As we'll later see, the dot product function on the gpu seems to ignore this flag
assert np.allclose(x_cpu_native, x_gpu_nonnative.get())

# Chenck endianness
# CPU
assert not x_cpu_nonnative.dtype.isnative
assert x_cpu_native.dtype.isnative
# GPU
assert not x_gpu_nonnative.dtype.isnative
assert x_gpu_native.dtype.isnative
# Fetching from GPU to CPU
assert not x_gpu_nonnative.get().dtype.isnative
assert x_gpu_native.get().dtype.isnative

# Dot product
# CPU
Ax_cpu_nonnative = A_cpu.dot(x_cpu_nonnative)
Ax_cpu_native = A_cpu.dot(x_cpu_native)
# GPU
Ax_gpu_nonnative = A_gpu.dot(x_gpu_nonnative)
Ax_gpu_native = A_gpu.dot(x_gpu_native)

# Print variables
print('A_cpu:')
print(A_cpu)
print('x_cpu_native: ', x_cpu_native)
print('Ax_cpu_native: ', Ax_cpu_native)
print('Ax_cpu_nonnative: ', Ax_cpu_nonnative)
print('Ax_gpu_native: ', Ax_gpu_native)
print('Ax_gpu_nonnative: ', Ax_gpu_nonnative)

# Compare
# After byte-swapping, the dot products on the cpu remain the same
assert np.allclose(Ax_cpu_nonnative, Ax_cpu_native)
# The dot product on the cpu and the dot product of the native version on the gpu is also the same
assert np.allclose(Ax_cpu_nonnative, Ax_gpu_native.get())
# However, when the nonnative version is used on the gpu, dot product returns the wrong answer
assert np.allclose(Ax_cpu_nonnative, Ax_gpu_nonnative.get())

Error messages, stack traces, or logs
A_cpu:
[[0. 1. 2.]
 [3. 4. 5.]
 [6. 7. 8.]]
x_cpu_native:  [0. 1. 2.]
Ax_cpu_native:  [ 5. 14. 23.]
Ax_cpu_nonnative:  [ 5. 14. 23.]
Ax_gpu_native:  [ 5. 14. 23.]
Ax_gpu_nonnative:  [3.044976e-319 1.217042e-318 2.129586e-318]

Traceback (most recent call last):
  File "endianness.py", line 50, in <module>
    assert np.allclose(Ax_cpu_nonnative, Ax_gpu_nonnative.get())
AssertionError

cupy.array is printed after it's sent to CPU.  https://github.com/cupy/cupy/blob/49af1c205bb99c86bdcb97d99f731589299ad86b/cupy/core/core.pyx#L1347-L1353
The following confirms cupy.ndarray.__mul__ ignores endian.
>>> x_cpu = np.arange(3.0).byteswap().newbyteorder()
>>> np.frombuffer(x_cpu, np.uint8).reshape(3, 8)
array([[  0,   0,   0,   0,   0,   0,   0,   0],
       [ 63, 240,   0,   0,   0,   0,   0,   0],
       [ 64,   0,   0,   0,   0,   0,   0,   0]], dtype=uint8)
>>> x_gpu = cp.array(x_cpu)
>>> x_gpu.dtype
dtype('>f8')
>>> np.frombuffer(x_gpu.tobytes(), np.uint8).reshape(3, 8)
array([[  0,   0,   0,   0,   0,   0,   0,   0],
       [ 63, 240,   0,   0,   0,   0,   0,   0],
       [ 64,   0,   0,   0,   0,   0,   0,   0]], dtype=uint8)
>>> y_gpu = x_gpu * 2
>>> np.frombuffer(y_gpu.tobytes(), np.uint8).reshape(3, 8)
array([[  0,   0,   0,   0,   0,   0,   0,   0],
       [126, 224,   1,   0,   0,   0,   0,   0],
       [128,   0,   0,   0,   0,   0,   0,   0]], dtype=uint8)

Let me close this and continue further discussion in #3652.


HIS Comment Density

If the following set of codecheck scripts is combined into a new python version, they should probably be updated to the new RatioCommentsWithBeforeToCode metric I just added. Metrics are available from the API even if not enabled. Even if the scripts are not combined, they should be checked for correct capitalization of the metric ids because metric ids are case sensitive.
AC_HIS_01
METRIC_02
HIS_01
It's probably also worth checking scripts/hisMetricsCsv.py

@jordancolbeth Would rewrite HIS_01 as a Python script that uses this new metric and see if that resolves the issue with Sophie?

This is done with 128f0300ff96365dabee47279645cf103326b9df. I'll follow up with Sophie and have her test it.

hisMetricsCsv.py is fixed with c35f7abf08e46f8ae398698e091de623db028639 as well


Feature : Added SearchProvider for Algolia

Addresses Issue #688

Thanks for the contribution @Yash-sudo-web Did you test this with Swirl?

Hey @dnicodemus, unfortunately, Swirl isn't supported for local installation or production on Windows, so I couldn't run it locally. However, I've thoroughly tested and validated the changes. Let me know if there's anything specific you'd like me to address or if you have any suggestions for a workaround. Thanks!

Hi @Yash-sudo-web , correct on the local use of Windows.  However, Swirl's Docker does run in Windows, and as you're change is new SearchProvider (vs. core code), you should be able to add it and test drive it via Docker.  Just remember that our docker does not persist any data on shutdown, so if you make changes to the SP config in Docker, be sure to copy them out somewhere if you need to preseve them.
Guidance for running Swirl in Docker (including on Windows):  https://github.com/swirlai/swirl-search/wiki/1.-Quick-Start#docker-installation

thanks @erikspears, I'll test the same and let you know

Hi @Yash-sudo-web, thanks for the PR. Please check the contributing guidelines. You need to create a PR to the develop branch by branching out from the develop branch.

Hey, sorry for the ignorance, I'll be closing this PR and create a new one following the contributing guidelines.


[expo plugin] fails social login and magic link login due to modification of immutable header

Is this suited for github?

[X] Yes, this is suited for github

To Reproduce

Create a backend with the expo() plugin
Attempt social login from the front-end.

Current vs. Expected behavior
Using the expo plugin on the server should work. Instead of working, a social login or magic login gives the following error message:
"TypeError: Can't modify immutable headers.",

This would be from the API/route: (it does the same thing for social login with google)
POST /api/auth/sign-in/magic-link

The culprit appears to be this line:
https://github.com/better-auth/better-auth/blob/main/packages/expo/src/index.ts#L28
I overrode that function (by writing my own plugin) with the following code:
    async onRequest(request) {
      // If the request has an "origin" header, do nothing
      if (request.headers.get("origin")) {
        return;
      }

      const expoOrigin = request.headers.get("expo-origin");
      // If there's no "expo-origin" header, do nothing
      if (!expoOrigin) {
        return;
      }

      // Clone the request to safely reuse the body
      const newHeaders = new Headers(request.headers);
      newHeaders.set("origin", expoOrigin);

      // Create a new Request object with updated headers and the cloned body
      const updatedRequest = new Request(request.clone() as Request<unknown, CfProperties<unknown>>, {
        headers: newHeaders,
      });

      console.log("created new request");
      return {
        request: updatedRequest,
      };
    },

This seemingly resolves the issue (though I still haven't gotten social or magic link login to work from the iOS simulator).
What version of Better Auth are you using?
1.1.4
Provide environment information
Server: CloudFlare Workers, Hono, Drizzle, D1.
Client: iOS simulator with Expo/React Native

Which area(s) are affected? (Select all that apply)
Backend
Auth config (if applicable)
betterAuth({
    secret: params.BETTER_AUTH_SECRET,
    advanced: {
      defaultCookieAttributes: {
        secure: true,
        sameSite: "none",
        path: "/",
      },
      useSecureCookies: false,
      cookieOptions: {
        sameSite: "none",
      },
    },
    user: {
      modelName: "users",
      additionalFields: {
        first_name: {
          type: "string",
          required: true,
          input: true,
        },
        last_name: {
          type: "string",
          required: true,
          input: true,
        },
        phone_number: {
          type: "string",
          required: false,
          input: true,
        },
      },
    },
    plugins: [
      expo(),
      magicLink({
        sendMagicLink: async ({ email, token, url }, request) => {
          try {
            const sesClient = new SESClient({
              region: params.AWS_DEFAULT_REGION || "us-east-1",
              credentials: {
                accessKeyId: params.AWS_ACCESS_KEY_ID || "",
                secretAccessKey: params.AWS_SECRET_ACCESS_KEY || "",
              },
            });

            const subject = "Your Magic Link";
            const fromAddress =<EMAIL_ADDRESS>
            // Log the magic link information
            log(Level.INFO, "Sending magic link", {
              email,
              token,
              url,
            });

            // Use sendSignupEmail to send the email
            await sendSignupEmail({
              sesClient,
              toAddress: email,
              subject,
              fromAddress,
              url,
            });

            console.log(
              `Magic link email sent to ${email} with token ${token}`,
            );
          } catch (error) {
            console.error("Error sending magic link:", error);
            throw new Error("Failed to send magic link");
          }
        },
      }),
    ],
    trustedOrigins: ["http://localhost:8081", "exp://"],
    socialProviders: {
      google: {
        clientId: params.GOOGLE_CLIENT_ID,
        clientSecret: params.GOOGLE_CLIENT_SECRET,
      },
    },
    database: drizzleAdapter(drizzle(db, { schema }), {
      provider: "sqlite",
      schema: {
        ...schema,
        user: schema.users,
      },
      usePlural: true,
    }),
    emailAndPassword: {
      enabled: false,
    },
  });

Additional context
That is all

+1
I have experienced the same issue with pretty much the same environment, with the only change being that I use a Postgres DB. Even after cloning the request I get either Please restart the process error or Internal server error from the cloudflare worker and I am not redirected back to the app, @brookewaters are you experiencing similar issues?
Currently I am using expo-apple-authentication and @react-native-google-signin/google-signin and logging in using id token authentication to get around this.


&TfLiteTensor->Data not 16-Byte Aligned

1. System information

OS Platform and Di_pywrap_toco_apistribution (e.g., Linux Ubuntu 16.04): LINUX UBUNTU 20.04
TensorFlow installation (pip package or built from source):  PIP PACKAGE
TensorFlow library (version, if pip package or github SHA, if built from source): PIP PACKAGE

2. Code
I think the code will not help, as the conversion process is working but I have questions about the resulting model file.
I have attached a zip:
model.zip
3. Failure after conversion
I am working on a proof of concept, and I am making changes to the TFLite codebase.
Although the model file is 16-byte aligned, some of the tensor data is not.
For example:
my input->data.f has an address of 0x..........0 (16 byte aligned)
my bias->data.f has an address of 0x...........8 (8 byte aligned)
While this can still work, the hardware optimization I am trying to implement works better if the data is 16-byte aligned. The only option I see is to specifically treat the first two floats (8 bytes), and then use full optimization for the rest of the data.
SO, my question is whether there is some option somewhere during conversion to align the the data within each tensor! I assume there would be some option to have it pad the data structures in the flatbuffer model with a few zeros before beginning the float data array.
The only posts I have seen regarding memory alignment are for TensorflowLite Micro, which isn't really an option for me.
5. (optional) Any other info / logs
If there is no option in the converter to achieve what I want, I suppose I can make the changes myself when the memory is allocated in TFlite, but I could use a nudge in the right direction. My assumption is that the model is parsed somewhere, and I would like to add some instruction to make sure the TfLiteTensor->data is properly aligned. Do you know where the easiest place to make that change would be?
Thanks!

For your specific cases, you might need to create your own data type in TensorFlow Lite and modify the converter and runtime to suppport it.

@srjoglekar246 could you take a look?

Hello, Sachin.
If it helps, you can load the model I provided into a debug version of Tensorflow Lite, and set a breakpoint at optimized_ops.h line 420 or so.  The PoC I am demonstrating branches from here. Two of the data arrays should be 16-byte aligned, and two should be 8-byte aligned, in the FullyConnected( ) function somewhere..  Again, while I can work around it, it would be preferable if there was some way to build/load the model in a way that all the tensor data arrays were 16-byte aligned by default.
Thanks in advance!

@MisterBerg you are correct, AFAIK the constants in TFLite flatbuffers (and even the runtime TFLite tensors) are required to be aligned at 8 bytes. This is mainly because most of the underlying backends/kernels we use don't require any greater alignment.
@miaout17 and I had discussed this a while back....YC, is there a way to make the converter align constants to a different multiple of 8 bytes? We cannot use an arbitrary alignment for obvious reasons, but any multiple of 64 bits should be okay I think.

Hi,
Thank you for opening this issue. Since this issue has been open for a long time, the code/debug information for this issue may not be relevant with the current state of the code base.
The TFLite team is constantly improving the framework by fixing bugs and adding new features. We suggest you try the latest TensorFlow version with the latest compatible hardware configuration which could potentially resolve the issue. If you are still facing the issue, please create a new GitHub issue with your latest findings, with all the debugging information which could help us investigate.
Please follow the release notes to stay up to date with the latest developments which are happening in the TFLite space.
Thanks.


Include context menu items from other extensions

Other extensions add new items to the context menu for files and folders in solution explorer. Those are missing inside of the File Explorer node.
For example, I tried to open a folder that is not included in the solution in Visual Studio Code. Usually I would do this using the "Open in Visual Studio Code" extension by Mads Kristensen. Sadly, this menu item is missing in the context menu in File Explorer.
I don't know if this has to be solved in this extension or in the ones adding the menu items. Hopefully, it is solved here once and for all, because I don't think every extension author would go to that length and add the menu items here as well.
Perhaps, the default context menu for files and folders in the solution explorer can be utilized here as well. This would solve my issue and provide access to further built-in functionality like the new comparison feature in VS.

It would also be beneficial to have Git (or Source control in general) menu items to compare versions/view file history.
It would be especially beneficial as there is currently no (known to me) option to view project's csproj file history in VS in Git. Checking the history by right clicking on the project will show the whole's project folder history, not only the file changes.
There is Source Control Explorer for TFS and this extension could help patch this hole :).


How can i use it with close source model. Gpt4o, 1.5 sonnet

You can now do it via OpenRouter using this fork: https://github.com/timothelaborie/MoA-Openrouter/blob/main/gui.ipynb


HDDS-10079. Remove validation of test method in flaky-test-check

What changes were proposed in this pull request?
If flaky-test-check is triggered for a specific method, it checks if the method exists in the test class.  This does not work for test methods inherited from parent class.
Test method testValidateBlockLengthWithCommitKey does not exist in TestSecureOzoneRpcClient.java.Stopping!

https://github.com/adoroszlai/ozone/actions/runs/7431424440/job/20222159860#step:3:33
However, the same test method is accepted for the parent class, which cannot be run, since it's abstract.
I propose removing this safety check.  If the specified method cannot be run, build output will show Tests run: 0.
https://issues.apache.org/jira/browse/HDDS-10079
How was this patch tested?
https://github.com/adoroszlai/ozone/actions/runs/7431422827

If the specified method cannot be run, build output will show Tests run: 0.

I guess the job would still pass though giving a false sense of success to user. I'm okay with current change given the parent class problem, but this would mean we need to check thoroughly when someone posts a successful job link. Should we fail if no tests are run?

If the specified method cannot be run, build output will show Tests run: 0.

I guess the job would still pass though giving a false sense of success to user. I'm okay with current change given the parent class problem, but this would mean we need to check thoroughly when someone posts a successful job link. Should we fail if no tests are run?

You are right, the job would still pass.  I've filed HDDS-10080 to fail fast (after first iteration) in junit.sh if no tests were run.
BTW, this can happen even prior to this patch, e.g. if I try to run TestOzoneRpcClientAbstract.

Should we fail if no tests are run?

This is implemented in #6036.

@sadanand48 with HDDS-10080 implemented, is this good to go?

Thanks @sadanand48 for the review.


"buffer size too large" when I send binary data, why?

"buffer size too large" when I send binary data, why?

bollo:~/d/seasocks (master|✔) $ git grep -i "buffer size too"
src/main/c/Connection.cpp:            LS_WARNING(_logger, "Closing connection: buffer size too large ("
bollo:~/d/seasocks (master|✔) $ grep -i "buffer size" -B4 src/main/c/Connection.cpp
        size_t endOfBuffer = _outBuf.size();
        size_t newBufferSize = endOfBuffer + bytesToBuffer;
        if (newBufferSize >= _server.clientBufferSize()) {
            LS_WARNING(_logger, "Closing connection: buffer size too large ("

Looks like the data you're sending is larger than the client's buffer size. That's controlled by...
bollo:~/d/seasocks (master|✔) $ git grep clientBufferSize
src/main/c/Connection.cpp:        if (newBufferSize >= _server.clientBufferSize()) {
src/main/c/Connection.cpp:                                    << newBufferSize << " >= " << _server.clientBufferSize() << ")");
src/main/c/Connection.cpp:    if (_request->contentLength() > _server.clientBufferSize()) {
src/main/c/Server.cpp:          _clientBufferSize(DefaultClientBufferSize),
src/main/c/Server.cpp:    _clientBufferSize = bytesToBuffer;
src/main/c/seasocks/Server.h:    size_t clientBufferSize() const override {
src/main/c/seasocks/Server.h:        return _clientBufferSize;
src/main/c/seasocks/Server.h:    size_t _clientBufferSize;
src/main/c/seasocks/ServerImpl.h:    virtual size_t clientBufferSize() const = 0;
src/test/c/MockServerImpl.h:    size_t clientBufferSize() const override {

looks like it's something that can be set in the Server.cpp....
bollo:~/d/seasocks (master|✔) $ grep -B4 '_clientBufferSize = bytesToBuffer' src/main/c/Server.cpp
}

void Server::setClientBufferSize(size_t bytesToBuffer) {
    LS_INFO(_logger, "Setting client buffer size to " << bytesToBuffer << " bytes");
    _clientBufferSize = bytesToBuffer;

looks like you can call server.setClientBufferSize(size) and increase the size of the buffer used to send data to the client.
The data is buffered as Seasocks tries to guarantee all the data makes it to the other side of the connection. It buffers the data and continues to send it even if the client on the end of the connection is slow and can't receive it all at once.
Hope that helps!


OrderedPartitionableRecord buffer full, storing iterator and retrying

Today I've got very strange issue (not a first time actually) when all middlemanagers stopped to consume data from Kinesis stream for a long period (>40 minutes).
The only message that was printed to tasks logs are these:
2019-12-10T21:36:22,218 INFO [appenderator_persist_0] org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver - Successfully handed off [1] segments.
2019-12-10T21:36:27,100 WARN [KinesisRecordSupplier-Worker-18] org.apache.druid.indexing.kinesis.KinesisRecordSupplier - OrderedPartitionableRecord buffer full, storing iterator and retrying in [5,000ms]
.............
2019-12-10T21:40:21,249 WARN [KinesisRecordSupplier-Worker-13] org.apache.druid.indexing.kinesis.KinesisRecordSupplier - OrderedPartitionableRecord buffer full, storing iterator and retrying in [5,000ms]
2019-12-10T21:40:21,500 INFO [intermediary-data-manager-0] org.apache.druid.indexing.worker.IntermediaryDataManager - Discovered partitions for [0] new supervisor tasks under location[var/tmp/intermediary-segments]
2019-12-10T21:40:21,503 INFO [intermediary-data-manager-0] org.apache.druid.indexing.worker.IntermediaryDataManager - Found [0] expired supervisor tasks
.............
2019-12-10T21:50:39,061 WARN [HttpClient-Netty-Worker-3] org.apache.druid.java.util.http.client.pool.ResourcePool - Resource at key[http://prod-druid-overlord-0.prod-druid-overlord-headless.prod-druid.svc.cluster.local:8090] was returned multiple times?
2019-12-10T21:50:39,061 WARN [task-runner-0-priority-0] org.apache.druid.discovery.DruidLeaderClient - Request[http://prod-druid-overlord-0.prod-druid-overlord-headless.prod-druid.svc.cluster.local:8090/druid/indexer/v1/action] failed.
org.jboss.netty.handler.timeout.ReadTimeoutException
	at org.jboss.netty.handler.timeout.ReadTimeoutHandler.<clinit>(ReadTimeoutHandler.java:84) ~[netty-3.10.6.Final.jar:?]
	at org.apache.druid.java.util.http.client.NettyHttpClient.go(NettyHttpClient.java:172) ~[druid-core-0.16.0-incubating.jar:0.16.0-incubating]
	at org.apache.druid.java.util.http.client.AbstractHttpClient.go(AbstractHttpClient.java:33) ~[druid-core-0.16.0-incubating.jar:0.16.0-incubating]
	at org.apache.druid.discovery.DruidLeaderClient.go(DruidLeaderClient.java:173) ~[druid-server-0.16.0-incubating.jar:0.16.0-incubating]
	at org.apache.druid.discovery.DruidLeaderClient.go(DruidLeaderClient.java:145) ~[druid-server-0.16.0-incubating.jar:0.16.0-incubating]
	at org.apache.druid.query.lookup.LookupReferencesManager.fetchLookupsForTier(LookupReferencesManager.java:569) ~[druid-server-0.16.0-incubating.jar:0.16.0-incubating]
	at org.apache.druid.query.lookup.LookupReferencesManager.tryGetLookupListFromCoordinator(LookupReferencesManager.java:422) ~[druid-server-0.16.0-incubating.jar:0.16.0-incubating]
	at org.apache.druid.query.lookup.LookupReferencesManager.lambda$getLookupListFromCoordinator$4(LookupReferencesManager.java:400) ~[druid-server-0.16.0-incubating.jar:0.16.0-incubating]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86) ~[druid-core-0.16.0-incubating.jar:0.16.0-incubating]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:114) ~[druid-core-0.16.0-incubating.jar:0.16.0-incubating]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:104) ~[druid-core-0.16.0-incubating.jar:0.16.0-incubating]
	at org.apache.druid.query.lookup.LookupReferencesManager.getLookupListFromCoordinator(LookupReferencesManager.java:390) ~[druid-server-0.16.0-incubating.jar:0.16.0-incubating]
	at org.apache.druid.query.lookup.LookupReferencesManager.getLookupsList(LookupReferencesManager.java:367) ~[druid-server-0.16.0-incubating.jar:0.16.0-incubating]
	at org.apache.druid.query.lookup.LookupReferencesManager.loadAllLookupsAndInitStateRef(LookupReferencesManager.java:350) ~[druid-server-0.16.0-incubating.jar:0.16.0-incubating]
	at org.apache.druid.query.lookup.LookupReferencesManager.start(LookupReferencesManager.java:156) ~[druid-server-0.16.0-incubating.jar:0.16.0-incubating]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_202]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_202]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_202]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_202]
	at org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:442) ~[druid-core-0.16.0-incubating.jar:0.16.0-incubating]
	at org.apache.druid.java.util.common.lifecycle.Lifecycle.start(Lifecycle.java:339) ~[druid-core-0.16.0-incubating.jar:0.16.0-incubating]
	at org.apache.druid.guice.LifecycleModule$2.start(LifecycleModule.java:140) ~[druid-core-0.16.0-incubating.jar:0.16.0-incubating]
	at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:115) ~[druid-services-0.16.0-incubating.jar:0.16.0-incubating]
	at org.apache.druid.cli.CliPeon.run(CliPeon.java:273) ~[druid-services-0.16.0-incubating.jar:0.16.0-incubating]
	at org.apache.druid.cli.Main.main(Main.java:113) ~[druid-services-0.16.0-incubating.jar:0.16.0-incubating]
.............
2019-12-10T22:18:28,614 WARN [KinesisRecordSupplier-Worker-19] org.apache.druid.indexing.kinesis.KinesisRecordSupplier - OrderedPartitionableRecord buffer full, storing iterator and retrying in [5,000ms]
2019-12-10T22:18:29,343 INFO [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver - New segment[midas_2019-12-10T21:00:00.000Z_2019-12-10T22:00:00.000Z_2019-12-10T21:00:04.939Z_54] for row[MapBasedInputRow{timestamp=2019-12-10T21:35:33.144Z, event={app_id=............

Affected Version
0.16.0
Description
Please include as much detailed information about the problem as possible.

Cluster size
8 middlemanagers x 2 peons
1 overlord
1 coordinator

Steps to reproduce the problem
happens sporadically, no issues reported from kinesis side (no throttling or other issues)

Relevant Kinesis metrics:

We are also facing this issue with v0.22.1 and the message OrderedPartitionableRecord buffer full, storing iterator and retrying in [5,000ms] shows up repeatedly.

We got this issue as well on 0.22.1
The root cause is that Coordinator stopped being responsive and various Polling loop starts throwing errors.

This problem only exists if basic-security module is turned on.

Looking at the 3 comments about issues here (from @didip, @applike-ss, & @dene14) it seems to me that the issues are probably related but different.
The original report by @dene14 has a really confusing log file. There is a message that a call to http://prod-druid-overlord-0.prod-druid-overlord-headless.prod-druid.svc.cluster.local:8090/druid/indexer/v1/action failed. But the following stack trace looks like a failed startup (note the Lifecycle.start). It should have come at the beginning of the log. And, at any rate, the stack trace is about a Coordinator API (note the LookupReferencesManager.fetchLookupsForTier: that's hitting the Coordinator) not an Overlord API. I wonder if the log got chopped up or went out-of-order somehow. The Kinesis metric timestamps also do not match the log timestamps, so it's hard to correlate these. It's been a long time since this report was filed, so I guess all the stuff required to debug it is long gone. This is unfortunate.
@applike-ss I'm interested in more information from task log files, if you have it. Kinesis metrics would be useful too. If they're in a different time zone from the log files, please let us know.
@didip Are you saying what happens is something like this?

The Coordinator becomes unresponsive for some reason.
Some time later, a new Kafka or Kinesis task starts up.
The task can't finish starting up because the Coordinator is unresponsive.
Ingestion falls behind, because the tasks can't start up.

Or are you saying that there is an issue where tasks that started up suddenly become unresponsive if the Coordinator has a problem?
Basically, I'm asking because I would expect the first case to happen: new processes (including tasks) can't start up if the Coordinator is unavailable and basic security is enabled. This is because they need to sync the user database from the Coordinator.
But, I wouldn't expect the second case to happen. If the Coordinator is unresponsive for some time while a task is running, it shouldn't prevent the task from making progress, unless the task is at a point where it needs to access some Coordinator API. Then, the task should basically sit there and wait for the Coordinator to come back.

Hi Gian!
First of all, I want to clarify, my problem revolves around batched
ingestion. But it's almost the exact same stack trace.

Or are you saying that there is an issue where tasks that started up
suddenly become unresponsive if the Coordinator has a problem?

From the perspective of batch ingestion, it's actually both. 97% of the
time when Peon starts up it's fine. But when you have 1-2 thousands of
Peons, a few of them will fail on boot when Coordinator becomes
unresponsive.
Unfortunately that's enough to bombed a few subtasks which in-turned bombed
the entire index_parallel.
I see that you already replied to me on Slack, we can continue the
conversation there and later on write up the solutions here.
On Fri, Jun 10, 2022 at 6:54 PM Gian Merlino @.***>
wrote:

Looking at the 3 comments about issues here (from @didip
https://github.com/didip, @applike-ss https://github.com/applike-ss,
& @dene14 https://github.com/dene14) it seems to me that the issues are
probably related but different.
The original report by @dene14 https://github.com/dene14 has a really
confusing log file. There is a message that a call to
http://prod-druid-overlord-0.prod-druid-overlord-headless.prod-druid.svc.cluster.local:8090/druid/indexer/v1/action
failed. But the following stack trace looks like a failed startup (note
the Lifecycle.start). It should have come at the beginning of the log.
And, at any rate, the stack trace is about a Coordinator API (note the
LookupReferencesManager.fetchLookupsForTier: that's hitting the
Coordinator) not an Overlord API. I wonder if the log got chopped up or
went out-of-order somehow. The Kinesis metric timestamps also do not match
the log timestamps, so it's hard to correlate these. It's been a long time
since this report was filed, so I guess all the stuff required to debug it
is long gone. This is unfortunate.
@applike-ss https://github.com/applike-ss I'm interested in more
information from task log files, if you have it. Kinesis metrics would be
useful too. If they're in a different time zone from the log files, please
let us know.
@didip https://github.com/didip Are you saying what happens is
something like this?

The Coordinator becomes unresponsive for some reason.
Some time later, a new Kafka or Kinesis task starts up.
The task can't finish starting up because the Coordinator is
unresponsive.
Ingestion falls behind, because the tasks can't start up.

Or are you saying that there is an issue where tasks that started up
suddenly become unresponsive if the Coordinator has a problem?
Basically, I'm asking because I would expect the first case to happen: new
processes (including tasks) can't start up if the Coordinator is
unavailable and basic security is enabled. This is because they need to
sync the user database from the Coordinator.
But, I wouldn't expect the second case to happen. If the Coordinator is
unresponsive for some time while a task is running, it shouldn't prevent
the task from making progress, unless the task is at a point where it needs
to access some Coordinator API. Then, the task should basically sit there
and wait for the Coordinator to come back.
—
Reply to this email directly, view it on GitHub
https://github.com/apache/druid/issues/9011#issuecomment-1152832935, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAARZVS25HON34FVMFCR3WTVOPWWFANCNFSM4JZGF2DA
.
You are receiving this because you were mentioned.Message ID:
@.***>

I think I see why the original log from @dene14 was so strange. The ReadTimeoutException is a liar! It is a static that is initialized once with a stack trace that matches whatever happened to first cause the class to get loaded up. But it
To make our logs more readable, we are going to need to wrap this. I raised a PR here: https://github.com/apache/druid/pull/12635

@gianm i will try to provide the logs once it happens again.

@applike-ss could you please share how many open segments are present for a given task by searching for occurrences of "Announcing segment"?
Was this stream resharded within 24 hours of this task's run? Are there any WARN / ERROR logs or any errors related to AWS for this stream in the overlord's logs during this time?

Hi, at least the question about the number of segments I could answer right away: We have roughly 21 open segments per task. We're using daily segment granularity and are getting data for the last ~21 days.

@applike-ss

The number of open segments may sometimes lead to a slower rate of ingestion than the rate at which records are consumed from kinesis. If a coarser granularity works, we could try changing a few values in the tuning config to see if it helps.
I was referring to the log OrderedPartitionableRecord buffer full, storing iterator and retrying

Thanks for explaining! The marked part is where the issues start. As you can see it happens for hours, but before that it's fine for a day in this screenshot. So my assumption is that our setup in general would be fast enough (also with the amount of segments). However it may very well be that i lack some knowledge that makes my statement invalid and in that case i will happily listen to your feedback :-)

@applike-ss, it does seem unlikely that it is because of open segments. Could you please help with a few more details?

Is the shard shardId-000000000056 present in any of the logs before this period [25/07 00:00]? When is its earliest occurrence?
Do the supervisor exceptions occur around the same time? [25/07 00:00]
The stacktrace corresponding to the Exception in supervisor run loop for dataSource logs would be very helpful as well.

yes, at least 5 days before that
The first OrderedPartitionableRecord error message happened at "2022-07-25 00:08:09.389", the first exception with message Exception in supervisor run loop for dataSource was at "2022-07-25 00:27:05.724"
Here you go:

java.util.concurrent.TimeoutException: Timeout waiting for task.
	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:269)
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96)
	at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.checkTaskDuration(SeekableStreamSupervisor.java:2618)
	at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.runInternal(SeekableStreamSupervisor.java:1270)
	at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor$RunNotice.handle(SeekableStreamSupervisor.java:333)
	at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.lambda$tryInit$3(SeekableStreamSupervisor.java:952)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

We MIGHT have found out what was causing this on our side.
It's still maybe a bit too early to celebrate, but we haven't had issues since quite some days now.
We were using a logging configuration that was not exactly elegant.
Our log4j was configured to use the HttpAppender to write logs to our log router.
This from time to time could fail due to network issues and resulted in an http exception within log4j.
Since we adjusted our strategy here, we haven't had a single hickup related to data ingestion.
Maybe this helps the original issue author as well?


Can't browse from the Azure IIoT Platform 2.7.11

Describe the bug
I have deployed the Azure Industrial IoT Platform as described here.
Afterwards I set up my onw Industrial IoT Gateway as described here on a Ubuntu 18.04.3 LTS.
When the GW was up and running I was able to see it in the Engineering tool under the Discovery menu. I configured the discovery and found the OPC UA Server to which I wanted to connect.
From the Browse menu I connected to the correct endpoint.
But when I click on the name of the endpoint nothing happens.
I checked the logs of the front-end and I can see the following message:
[2020-04-20T13:44:59.977Z] Information: Normalizing '_blazor' to 'https://softing-edge-iiot.azurewebsites.net/_blazor'.	blazor.server.js:1
[2020-04-20T13:45:01.592Z] Information: WebSocket connected to wss://softing-edge-iiot.azurewebsites.net/_blazor?id=o6NIOwh7EfokwkRLO9h4ag. blazor.server.js:15
[2020-04-20T13:45:13.731Z] Error: There was an unhandled exception on the current circuit, so this circuit will be terminated. For more details turn on detailed exceptions by setting 'DetailedErrors: true' in 'appSettings.Development.json' or set 'CircuitOptions.DetailedErrors'.
	e.log @ blazor.server.js:15
	C @ blazor.server.js:8
	(anonymous) @ blazor.server.js:8
	(anonymous) @ blazor.server.js:1
	e.invokeClientMethod @ blazor.server.js:1
	e.processIncomingData @ blazor.server.js:1
	connection.onreceive @ blazor.server.js:1
	i.onmessage @ blazor.server.js:1
	blazor.server.js:1
[2020-04-20T13:45:13.732Z] Information: Connection disconnected.

To Reproduce
Steps to reproduce the behavior:

Start an IIoT Gateway
Discover the servers from the Engineering Tool
Connect to the correct endpoint
Try to browse the connected endpoint.

Expected behavior
I would expect to be able to browse the endpoint.
Additional context
After this problem occures I can't interact with the webpage. I can switch between the menus but other than the loading spinner no content is displayed. It seems that the websocket connection is closed and it's not opened again. The page seems to recover after a Hard Reload, but the browsing still doesn't work.
There doesn't seem to be any error in either of the edge modules (discovery, opctwin, opcpublisher) I'm assuming this is a webapp/frontend issue.

In some cases after the failed browse attempt, when reloading the page a Internal Server Error page appears, like the one below:

The exact message of the error is this:
System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.Azure.IIoT.App.Pages.Browser.BuildRenderTree(RenderTreeBuilder __builder)
   at Microsoft.AspNetCore.Components.Rendering.ComponentState.RenderIntoBatch(RenderBatchBuilder batchBuilder, RenderFragment renderFragment)
   at Microsoft.AspNetCore.Components.RenderTree.Renderer.RenderInExistingBatch(RenderQueueEntry renderQueueEntry)
   at Microsoft.AspNetCore.Components.RenderTree.Renderer.ProcessRenderQueue()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Components.Rendering.HtmlRenderer.HandleException(Exception exception)
   at Microsoft.AspNetCore.Components.RenderTree.Renderer.ProcessRenderQueue()
   at Microsoft.AspNetCore.Components.RenderTree.Renderer.AddToRenderQueue(Int32 componentId, RenderFragment renderFragment)
   at Microsoft.AspNetCore.Components.ComponentBase.StateHasChanged()
   at Microsoft.AspNetCore.Components.ComponentBase.CallOnParametersSetAsync()
   at Microsoft.AspNetCore.Components.ComponentBase.RunInitAndSetParametersAsync()
   at Microsoft.AspNetCore.Components.Rendering.HtmlRenderer.HandleException(Exception exception)
   at Microsoft.AspNetCore.Components.Rendering.ComponentState.SetDirectParameters(ParameterView parameters)
   at Microsoft.AspNetCore.Components.RenderTree.Renderer.RenderRootComponentAsync(Int32 componentId, ParameterView initialParameters)
   at Microsoft.AspNetCore.Components.Rendering.HtmlRenderer.CreateInitialRenderAsync(Type componentType, ParameterView initialParameters)
   at Microsoft.AspNetCore.Components.Rendering.HtmlRenderer.RenderComponentAsync(Type componentType, ParameterView initialParameters)
   at Microsoft.AspNetCore.Components.Rendering.RendererSynchronizationContext.<>c__11`1.<<InvokeAsync>b__11_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.ViewFeatures.StaticComponentRenderer.PrerenderComponentAsync(ParameterView parameters, HttpContext httpContext, Type componentType)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.ComponentRenderer.PrerenderedServerComponentAsync(HttpContext context, ServerComponentInvocationSequence invocationId, Type type, ParameterView parametersCollection)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.ComponentRenderer.RenderComponentAsync(ViewContext viewContext, Type componentType, RenderMode renderMode, Object parameters)
   at Microsoft.Azure.IIoT.App.Pages.Pages__Host.<ExecuteAsync>b__12_1() in D:\a\1\s\samples\src\Microsoft.Azure.IIoT.App\src\Pages\_Host.cshtml:line 19
   at Microsoft.AspNetCore.Razor.Runtime.TagHelpers.TagHelperExecutionContext.SetOutputContentAsync()
   at Microsoft.Azure.IIoT.App.Pages.Pages__Host.ExecuteAsync()
   at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageCoreAsync(IRazorPage page, ViewContext context)
   at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageAsync(IRazorPage page, ViewContext context, Boolean invokeViewStarts)
   at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderAsync(ViewContext context)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ViewContext viewContext, String contentType, Nullable`1 statusCode)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ViewContext viewContext, String contentType, Nullable`1 statusCode)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResultFilterAsync>g__Awaited|29_0[TFilter,TFilterAsync](ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResultExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)

In which service should I try to look into, what could be the source of the problem?

which server are you using?
From the debug tools of your browser could you send the console output?

This is the debug log from the browser:
[2020-05-07T08:08:01.444Z] Error: There was an unhandled exception on the current circuit, so this circuit will be terminated. For more details turn on detailed exceptions by setting 'DetailedErrors: true' in 'appSettings.Development.json' or set 'CircuitOptions.DetailedErrors'.		 blazor.server.js:15
	e.log	@	blazor.server.js:15
[2020-05-07T08:08:01.449Z] Information: Connection disconnected.			blazor.server.js:1

And I get the following in the Log Stream from the corresponding app service:
2020-05-07T08:07:48.406046538Z [08:07:48 INF] Authorization was successful.
2020-05-07T08:07:48.406191439Z [08:07:48 INF] Authorization was successful.
2020-05-07T08:07:48.406321840Z [08:07:48 INF] Authorization was successful.
2020-05-07T08:07:48.407026744Z [08:07:48 WRN] Unhandled exception rendering component: Object reference not set to an instance of an object.
2020-05-07T08:07:48.407044044Z System.NullReferenceException: Object reference not set to an instance of an object.
2020-05-07T08:07:48.407050544Z    at Microsoft.Azure.IIoT.App.Pages.Browser.BuildRenderTree(RenderTreeBuilder __builder)
2020-05-07T08:07:48.407064645Z    at Microsoft.AspNetCore.Components.Rendering.ComponentState.RenderIntoBatch(RenderBatchBuilder batchBuilder, RenderFragment renderFragment)
2020-05-07T08:07:48.407069645Z    at Microsoft.AspNetCore.Components.RenderTree.Renderer.RenderInExistingBatch(RenderQueueEntry renderQueueEntry)
2020-05-07T08:07:48.407074145Z    at Microsoft.AspNetCore.Components.RenderTree.Renderer.ProcessRenderQueue()
2020-05-07T08:07:48.407627748Z [08:07:48 ERR] Unhandled exception in circuit 'aZ-MEEPNVp5b4hmQxNXG51Ya6IMJpI6-pjdTOIwjWOU'.
2020-05-07T08:07:48.407641448Z System.NullReferenceException: Object reference not set to an instance of an object.
2020-05-07T08:07:48.407646748Z    at Microsoft.Azure.IIoT.App.Pages.Browser.BuildRenderTree(RenderTreeBuilder __builder)
2020-05-07T08:07:48.407651248Z    at Microsoft.AspNetCore.Components.Rendering.ComponentState.RenderIntoBatch(RenderBatchBuilder batchBuilder, RenderFragment renderFragment)
2020-05-07T08:07:48.407655648Z    at Microsoft.AspNetCore.Components.RenderTree.Renderer.RenderInExistingBatch(RenderQueueEntry renderQueueEntry)
2020-05-07T08:07:48.407660048Z    at Microsoft.AspNetCore.Components.RenderTree.Renderer.ProcessRenderQueue()

Closed as not reproducible with latest preview


Runtime Error: ImportError: No module named core

I build TASO from source according the install doc.
mkdir build; cd build; cmake ..
sudo make install -j 4
cd ../python
python setup.py install

the output info of  cmake ..  as below:
-- The CXX compiler identification is GNU 5.4.0
-- The CUDA compiler identification is NVIDIA 9.0.176
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc
-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc -- works
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Found Protobuf: /root/3rdparty/protobuf-3.9.0/lib/libprotobuf.so;-lpthread (found version "3.9.0")
-- PROTOBUF=/root/3rdparty/protobuf-3.9.0/lib/libprotobuf.so
-- Performing Test SUPPORT_CXX11
-- Performing Test SUPPORT_CXX11 - Success
-- Found Threads: TRUE
-- Found CUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda
-- Found CUDA_CUDA_LIBRARY=/usr/local/cuda/targets/x86_64-linux/lib/stubs/libcuda.so
-- Found CUDA_CUDART_LIBRARY=/usr/local/cuda/lib64/libcudart.so
-- Found CUDA_NVRTC_LIBRARY=/usr/local/cuda/lib64/libnvrtc.so
-- Found CUDA_CUDNN_LIBRARY=/usr/lib/x86_64-linux-gnu/libcudnn.so
-- Found CUDA_CUBLAS_LIBRARY=/usr/local/cuda/lib64/libcublas.so
-- CUDA_INCLUDE_DIR=/usr/local/cuda/include
-- Configuring done
-- Generating done
-- Build files have been written to: /mnt/tscpfs2/xiaotao.chen/Repositories/TASO/build

the output info of make install -j as below:
Install the project...
-- Install configuration: ""
-- Installing: /usr/local/lib/libtaso_runtime.so
-- Set runtime path of "/usr/local/lib/libtaso_runtime.so" to ""
-- Up-to-date: /usr/local/./include
-- Installing: /usr/local/./include/taso
-- Installing: /usr/local/./include/taso/ops.h
-- Installing: /usr/local/./include/taso/substitution.h
-- Installing: /usr/local/./include/taso/cuda_helper.h

**after run python setup.py install, to check taso is installed by pip list|grep taso **
taso                               0.1.0

Then i try to run the example with the script python examples/resnext50.py, and it shows:
Traceback (most recent call last):
  File "examples/resnext50.py", line 1, in <module>
    import taso as ts
  File "/mnt/tscpfs2/xiaotao.chen/Repositories/TASO/python/taso/__init__.py", line 1, in <module>
    from .core import *
ImportError: No module named core

I don't know which step i missed. May your help, Thanks. @jiazhihao

@XiaotaoChen I saw you have a new post. Have you fixed this issue?

@jiazhihao Thanks, That should be some errors in my envs. I rebuild a docker with your Dockerfile, It's work normally.


Runtime failure in package:test bootstrapping code with kernel/ddk

We can now successfully build the tests but we get runtime failures (in the tests themselves).
The failure looks like this (unfortunately we don't have source maps working yet):
Cannot set property Symbol(_scanner) of [object Object] which has only a getter
  package:string_scanner/string_scanner.ddc.js 865:22       new
  package:string_scanner/string_scanner.ddc.js 612:14       get state
  package:boolean_selector/boolean_selector.ddc.js 853:112  [_getNext]
  package:boolean_selector/boolean_selector.ddc.js 835:60   peek
  package:boolean_selector/boolean_selector.ddc.js 845:29   scan
  package:boolean_selector/boolean_selector.ddc.js 988:38   [_and]
  package:boolean_selector/boolean_selector.ddc.js 982:28   [_or]
  package:boolean_selector/boolean_selector.ddc.js 972:32   [_conditional]
  package:boolean_selector/boolean_selector.ddc.js 965:40   parse
  package:boolean_selector/boolean_selector.ddc.js 224:61   parse
  package:test_api/test_api.ddc.js 848:140                  <fn>
  package:test_api/test_api.ddc.js 778:32                   _wrapFormatException
  package:test_api/test_api.ddc.js 848:70                   parse
  package:test_api/test_api.ddc.js 1532:129                 deserialize
  package:test_api/src/remote_listener.ddc.js 116:28        <fn>
  dart:sdk_internal 24965:9

I am not sure if this is a dev_compiler or kernel issue, or an issue in the code which only surfaces on the web for some reason.

For repro instructions you can checkout the kernel-caching branch of the build repo (http://github.com/dart-lang/build), cd into the _test directory, and run pub run build_runner test -- -p chrome test/hello_world_test.dart.

It looks like a bug in the compiler, I'll investigate.

I can reproduce by running DDC -k on https://github.com/dart-lang/string_scanner/blob/master/test/span_scanner_test.dart

I can reproduce by running DDC -k on https://github.com/dart-lang/string_scanner/blob/master/test/span_scanner_test.dart

great, that should make it easier to diagnose hopefully :)

CFE is (correctly?) inserting a noSuchMethod stub for private getters such as _scanner in this example:
/// A class representing the state of a [SpanScanner].
class _SpanScannerState implements LineScannerState {
  /// The [SpanScanner] that created this.
  final RelativeSpanScanner _scanner;

  final int position;
  int get line => _scanner._sourceFile.getLine(position);
  int get column => _scanner._sourceFile.getColumn(position);

  _SpanScannerState(this._scanner, this.position);
}

The reason is that LineScannerState is in another library, so LineScannerState._scanner is not implemented here. Thus the nSM stub is correct, as long as its name correctly refers to its originating library, not the library with _SpanScannerState. I'll double check that.
DDC/K does not currently have the capability of referring to library-private names in another module, but I'm guessing that won't affect you because all of the package sources will be compiled together. I am putting together a fix for the cross module issue (library-private names will be exported, see https://github.com/dart-lang/sdk/issues/36252). The fix for that issue might fix this one too, I'll check.

Simple repro:
import 'test2.dart';
class C implements B {
  final int _x;
  C(this._x);
}

main() {
  var c = C(42);
  print(c._x);
}

// in test2.dart
class B {
  final int _x;
  B(this._x);
}

https://dart-review.googlesource.com/c/sdk/+/97441


custom location is not being auto populated causing notebook execution error while creating SQL Managed Instance (Arc Data services)

Azure Data Studio Version:

Version: 1.36.1 (system setup)
Commit: e56e1d931bbb110fe8f24751423c4e184bb8f22d
Date: 2022-04-21T21:25:10.399Z
VS Code: 1.59.0
Electron: 13.6.6
Chrome: 91.0.4472.164
Node.js: 14.16.0
V8: 9.1.269.39-electron.0

OS Version: Windows

OS: Windows_NT x64 10.0.20348

Executing: az --version

Successfully executed: az --version
>>>Output: azure-cli                         2.36.0
core                              2.36.0
telemetry                          1.0.6
Extensions:
arcdata                            1.4.0
connectedk8s                       1.2.8
customlocation                     0.1.3
k8s-extension                      1.2.0
Dependencies:
msal                              1.17.0
azure-mgmt-resource               20.0.0
Python location 'C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe'
Extensions directory 'C:\Users\arcdemo.azure\cliextensions'
Python (Windows) 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, 22:57:10) [MSC v.1929 32 bit (Intel)]
Legal docs and information: aka.ms/AzureCliLegal
Your CLI is up-to-date.
Steps to Reproduce\Issue:
While using Azure Data Studio generated notebook to create SQL Managed instance, custom location is not being auto populated which is causing notebook execution error:
arc_data_controller_connection_mode = 'direct'
arc_data_controller_location = 'eastus'
arc_data_controller_custom_location = ''
print('Variables have been set successfully.')
Error:
Command: az sql mi-arc create --name sqlinstance2 --resource-group Arc-Data-May --location eastus --custom-location  --replicas 1 --cores-request "2" --cores-limit "4" --memory-request "4Gi" --memory-limit "8Gi" --storage-class-data "default" --storage-class-datalogs "default" --storage-class-logs "default" --storage-class-backups "default" --volume-size-data 5Gi --volume-size-datalogs 5Gi --volume-size-logs 5Gi --volume-size-backups 5Gi --tier GeneralPurpose --dev --license-type BasePrice --cores-limit 4 failed

Error output: ERROR: argument --custom-location: expected one argument

Does this issue occur when all extensions are disabled?: NA

@candiceye


Decouple models from core library

Models could be decoupled from core library in order to permit code reuse when building .NET apps that consumes the BGPViewerOpenAPI

Models were decoupled in last pull request merged.

Models were decoupled in last pull request merged.


ARROW-13020: [C++] [Gandiva] Enable Gandiva to run the LLVM in Interpreted mode - WIP

LLVM Limitations

Looking at the LLVM Documentation and for some links. It seems that for the Interpreted mode, the LLVM does not support for some types and some functions that are used by the Gandiva and that work in JIT mode.
https://www.graalvm.org/reference-manual/llvm/NativeExecution/ (to find the references about the LLVM, search for LLVM instruction set support)


Possibility to set background color

Simple setter to private value used in onDraw to fill canvas with initial color.

@barteksc Can we get this merged?

I will add solution that was present in branch 1.x - canvas is drawn with drawable retrieved by View#getBackground()

Version 2.2.0 retrieves canvas background from View#getBackground(). But thank you for pull request.


is softmax required?

The model passes the logits through softmax in the output layer, but then the F.cross_entropy() function is used which combines log_softmax and nll. I think maybe the softmax in the output layer is not necessary?

Hi, it's been a little while since I looked at this code - but I'll take a look. Thanks for bringing it up!

Thanks @as595 for pointing this out, I'll fix it later today - or you're more than welcome to submit a PR.
Brief Documentation of the Issue:
It appears the softmax in the output layer is unnecessary:
https://github.com/danielkelshaw/ConcreteDropout/blob/7c0fdd79d13d88e61ee68b4d15269cfdf5d64043/mnist_example.py#L42
In the train() function, this is effectively getting used twice:
https://github.com/danielkelshaw/ConcreteDropout/blob/7c0fdd79d13d88e61ee68b4d15269cfdf5d64043/mnist_example.py#L57-L59
Code for the test() function will need to change slightly:
https://github.com/danielkelshaw/ConcreteDropout/blob/7c0fdd79d13d88e61ee68b4d15269cfdf5d64043/mnist_example.py#L78-L83
Predictions are based on the logits passed through softmax so the code will need to be amended to ensure the test_loss is calculated correctly.


Over-large message element crashes engine.

Due to the limited maximum size of the variable type (FNAME I think), if a message element is received which is larger than 1024 chars the engine crashes.
Ultimately we should think about changing the type for this message element type, but in the short term we should think about handling the error more gracefully.

I pushed the support for OSC blobs.
You should probably use them for use cases where long strings are needed.
It does not fix this issue though:

long OSC strings are not handled correctly (a fix would break compatibility with previous version)
it still crashes (a fix needs a strlen call which is "expensive")


[pyenv] handle the PATH for pyenv

this PR adds a ~/.profile file to the setup in order to handle the PATH manipulations for pyenv
context
pyenv 2.0.0 introduced around May 2021 a breaking change where the PATH is no longer handled by the pyenv initcommand
the solution proposed when running the pyenv init command is to handle the PATH for pyenv manually in the ~/.profile file (where the user custom PATH manipulations are supposed to take place, not in the ~/.zshrc file) :
# (The below instructions are intended for common
# shell setups. See the README for more guidance
# if they don't apply and/or don't work for you.)

# Add pyenv executable to PATH and
# enable shims by adding the following
# to ~/.profile and ~/.zprofile:

export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init --path)"

# Load pyenv into the shell by adding
# the following to ~/.zshrc:

eval "$(pyenv init -)"

# Make sure to restart your entire logon session
# for changes to profile files to take effect.

@krokrob looks like https://github.com/lewagon/dotfiles/blob/master/install.sh does that for us. I ran the script manually to verify and it seems to be the work correctly (almost). Basically it creates a symlink version of all the files at the root of dotfiles in the home of the user.
The following paragraph of the file reads (approximately) as follows afaik :

iterate on all the files in the dotfiles dir (install.sh is supposedly ran from the dotfiles dir)

if the file is not a directory

define target file path in the home dir as . + filename
if the target is not a .sh script or README.md or .sublime-settings

create a backup of the file with corresponding name + .backup in the home
☝ btw this part seems to fail as no .backup files are generated in my home as I successively run the script
then create a symlink to the file in the dotfiles (git controlled) from the corresp .file in the home dir

#!/bin/zsh
for name in *; do
  if [ ! -d "$name" ]; then
    target="$HOME/.$name"
    if [[ ! "$name" =~ '\.sh$' ]] && [ "$name" != 'README.md' ] && [[ ! "$name" =~ '\.sublime-settings$' ]]; then
      backup $target

      if [ ! -e "$target" ]; then
        echo "-----> Symlinking your new $target"
        ln -s "$PWD/$name" "$target"
      fi
    fi
  fi
done

I have not tried to bullet proof validate the whole thing - appart from the type -a pyenv > /dev/null && which makes sure that the students of the web dev bootcamp that do not have pyenv installed are not impacted by the setup, I was kind of hoping for a blind validation 😉 😁
No really someone should probably bullet proof this, is there someone who is already up to speed on this ?

ps : looks like rerunning install.sh does the trick for students having already completed the setup...

@krokrob looks like https://github.com/lewagon/dotfiles/blob/master/install.sh does that for us. I ran the script manually to verify and it seems to do the job correctly. Basically it creates a symlink to all the files at the root of dotfiles in the home of the user.
The following paragraph of the file reads (approximately) as follows afaik :
* iterate on all the files in the dotfiles dir (install.sh is supposedly ran from the dotfiles dir)

  * if the file is not a directory

    * define target file path in the home dir as . + filename
    * if the target is not a .sh script or README.md or .sublime-settings

      * create a backup of the file with corresponding name + .backup in the home
        ☝ only backup if the file is a pure file (not a symlink)
      * then create a symlink to the file in the dotfiles (git controlled) from the corresp .file in the home dir
        ☝ if the target does not exist - which should not be the case (unless you are evil an intentionally created a non symlink .file and a .file.backup previously to running the script)

#!/bin/zsh
for name in *; do
  if [ ! -d "$name" ]; then
    target="$HOME/.$name"
    if [[ ! "$name" =~ '\.sh$' ]] && [ "$name" != 'README.md' ] && [[ ! "$name" =~ '\.sublime-settings$' ]]; then
      backup $target

      if [ ! -e "$target" ]; then
        echo "-----> Symlinking your new $target"
        ln -s "$PWD/$name" "$target"
      fi
    fi
  fi
done

I have not tried to bullet proof validate the whole thing - appart from the type -a pyenv > /dev/null && which makes sure that the students of the web dev bootcamp that do not have pyenv installed are not impacted by the setup, I was kind of hoping for a blind validation 😉 😁
No really someone should probably bullet proof this (or agree to creating a new .profile file handling the PATH related operations - which could be extracted from the .zshrc ?), is there someone who is already up to speed on this ?

Crystal clear, thanks!


[Feature Request] Electron

What is your enhancement?
Add support to nextron or electron

Hello there, @inginheiiro ,
Unfortunately, we won't add support for nextron or electron.
Best,
Manu


Silenced checks do not show as acknowledged

In Sensu 0.26 the Silencing feature was overhauled to no longer use Stashes, as such when failing checks are now silenced within Sensu they are not appearing as acknowledged within Grid.
Silenced checks are now accessed via the /silenced API endpoint, example output below:
[
  {
    "id": "client:nonprodrabbit4:*",
    "subscription": "client:nonprodrabbit4",
    "check": null,
    "reason": null,
    "creator": "admin",
    "expire_on_resolve": true,
    "expire": 3272
  }
]

Would it please be possible to update Grid to look at this new endpoint so that it supports Sensu 0.26?
Cheers

+1
I would like this update too please.
Thanks,
Tom

hi guys, had a quick look and the changes weren't too much work to get this working.
I'll close this now but please do re-open if you think there's something else still not quite working right with the ACK count.
Thanks for bringing this up!
Alex