It's hard to debug connectivity_state errors...

What version of gRPC and what language are you using?
Grpc.Core, 1.19.0, C#.
What operating system (Linux, Windows,...) and version?
Windows 10
What runtime / compiler are you using (e.g. python version or version of gcc)
Visual Studio 2017, .NET Core 2.0. dotnet compiler version: 2.1.505
What did you do?
I set my API server endpoint to a non-existent endpoint. However, I cannot get the grpc client library to fail reliably.
Half the time, it fails with Status(StatusCode=Unavailable, Detail="Name resolution failure")
The other half, it fails with Status(StatusCode=Unavailable, Detail="channel is in state TRANSIENT_FAILURE")
The more troubling fact is that it doesn't give me additional useful debugging information. I have logging like this:
Environment.SetEnvironmentVariable("GRPC_VERBOSITY", "DEBUG");
Environment.SetEnvironmentVariable("GRPC_TRACE", "connectivity_state");

But when things fail, the logs don't tell me what failed. Is it a 404 on the URL? Flaky connection? Something else? The logs look like this instead.
D0401 14:58:01.610543 Grpc.Core.Internal.UnmanagedLibrary Attempting to load native library "C:\Users\anash\.nuget\packages\grpc.core\1.19.0\lib\netstandard1.5\../..\runtimes/win/native\grpc_csharp_ext.x64.dll"
D0401 14:58:01.647022 Grpc.Core.Internal.NativeExtension gRPC native library loaded successfully.
D0401 14:58:01.649159 0 T:\src\github\grpc\workspace_csharp_ext_windows_x64\src\core\ext\filters\client_channel\resolver\dns\c_ares\dns_resolver_ares.cc:488: Using ares dns resolver
I0401 14:58:01.810156 0 T:\src\github\grpc\workspace_csharp_ext_windows_x64\src\core\lib\transport\connectivity_state.cc:81: CONWATCH: 00000258EC5E7A10 request_router: get IDLE
I0401 14:58:01.902005 0 T:\src\github\grpc\workspace_csharp_ext_windows_x64\src\core\lib\transport\connectivity_state.cc:164: SET: 00000258EC5E7A10 request_router: IDLE --> TRANSIENT_FAILURE [resolver_result] error=00000258ED5CBE50 {"created":"@1554145081.901000000","description":"No load balancing policy","file":"T:\src\github\grpc\workspace_csharp_ext_windows_x64\src\core\ext\filters\client_channel\request_routing.cc","file_line":806}

Additional details: I have multiple users reporting https://github.com/googleads/google-ads-dotnet/issues/59, but the lack of good logging and the random nature of the error being thrown is preventing me from debugging things further.

I set my API server endpoint to a non-existent endpoint - note the channel will continuously retry name resolution, and it will wait for some time in between retries - in this way the error message between different operations may differ depending on the momentary state of the channel.
There are some environment variables that can be set to enable verbose tracing of the core library. See https://github.com/grpc/grpc/blob/master/doc/environment_variables.md, and specifically the GRPC_TRACE bullet.
For example, for this particular issue it will probably be useful to run the binary under these environment variables:
GRPC_VERBOSITY=DEBUG
GRPC_TRACE=api,http,cares_resolver,cares_address_sorting.
Also, for a name resolution error, it will be interesting to run the test under GRPC_DNS_RESOLVER=native and see if the error persists (see the GRPC_DNS_RESOLVER  bullet under https://github.com/grpc/grpc/blob/master/doc/environment_variables.md for more details on that). There have been DNS name resolver changes recently which might be related - see open issue https://github.com/grpc/grpc/issues/18411

@AnashOommen if you have a repro script and an error log, please share

Hi @apolcyn

Hi @apolcyn
I went to the link https://github.com/grpc/grpc/blob/master/doc/environment_variables.md and added the following logging code
Environment.SetEnvironmentVariable("GRPC_TRACE", "api,http,cares_resolver,cares_address_sorting");
Environment.SetEnvironmentVariable("GRPC_VERBOSITY", "debug");
Grpc.Core.GrpcEnvironment.SetLogger(new Grpc.Core.Logging.ConsoleLogger());
And the code worked and I was able to get the Campaign data.
But after some time again I started getting the exception of "Name Resolution Failure". I found it surprising as the code worked for just some time and again throwing exception.
What might be the reason behind this

Hi @apolcyn,
I suspect this is very machine-specific in nature. @Swapygit can replicate this on his machine, but not on his colleague's machine.
He is using Google Ads API, which needs additional credentials to work, and unfortunately I cannot share them online on this issue tracker. Perhaps, I can share my personal test account creds internally?

This is the repro script, right: https://github.com/googleads/google-ads-dotnet/issues/59#issuecomment-474778537?
I am trying to repro with that - I think I can use my own creds for it

BTW, @Swapygit , when you ran your code under:
Environment.SetEnvironmentVariable("GRPC_TRACE", "api,http,cares_resolver,cares_address_sorting");
Environment.SetEnvironmentVariable("GRPC_VERBOSITY", "debug");
Grpc.Core.GrpcEnvironment.SetLogger(new Grpc.Core.Logging.ConsoleLogger());

If you can share the output, it would be helpful in order to see how this is failing, so that we can try to diagnose something that only occurs on your machine.
BTW, suggest posting the log output with:
GRPC_TRACE=api,cares_resolver,cares_address_sorting
rather than GRPC_TRACE=api,http,cares_resolver,cares_address_sorting
because log output with http on may display your OAuth token.

@Swapygit I should also add, a workaround for the time being should be to run your application under
environment variables setting: GRPC_DNS_RESOLVER=native

Sorry to spam - it may also be worth trying with the 1.20.0-pre1 gRPC nuget package. Among other things, this includes an update to the c-ares DNS resolver library which is used by the 1.19 release (https://github.com/grpc/grpc/pull/18519)

Hi @apolcyn , @AnashOommen
The code worked when I added the GRPC_DNS_RESOLVER=native.
I have one more query. In adwords API we used to fetch ALL_TIME Campain_Performance_Report. But how will I fetch the same in google Ads Api as there is no Date range for "ALL_TIME" in google Ads Query language
Kindly help

@apolcyn is then fine to switch to native dns resolver in Ads client
library?
Did you get a chance to replicate the original issue I opened the thread
about?
@Swapygit let’s contie Ads API discussion on the other issue thread.
On Thu, Apr 4, 2019 at 8:21 AM Swapygit<EMAIL_ADDRESS>wrote:

Hi @apolcyn https://github.com/apolcyn , @AnashOommen
https://github.com/AnashOommen
The code worked when I added the GRPC_DNS_RESOLVER=native.
I have one more query. In adwords API we used to fetch ALL_TIME
Campain_Performance_Report. But how will I fetch the same in google Ads Api
as there is no Date range for "ALL_TIME" in google Ads Query language
Kindly help
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/grpc/grpc/issues/18597#issuecomment-479874444, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AETkiE7XLMhro7IQoXPZSR8wg2Dlg3SOks5vde5PgaJpZM4cWfhq
.

--
Anash P. Oommen | Developer Programs Engineer |<EMAIL_ADDRESS>|
+1-650-440-2854

Sorry the delayed response.

@apolcyn fine to switch to native dns resolver in Ads client
library?

I see no issue if users can run their applications under the GRPC_DNS_RESOLVER=native environment variable, if it fixes things. However, are you talking about switching to the "native" DNS resolver for the whole Ads client library?
Follow-up question: in doing so would the Ads library internally set the environment variable GRPC_DNS_RESOLVER=native somehow, or that would there just be a recommendation that users set the env variable themselves?

Did you get a chance to replicate the original issue I opened the thread
about?

By the original issue, I assume you mean the case where the channel targets a non-existant endpoint, and the error messages of RPC's flaps between "Name resolution failure" and "channel is in state transient failure".
If so, unfortunately there's not much that can be done about this - the channel doesn't know that it will never successfully resolve the endpoint, and so it will repeatedly retry in doing so, with backoff time intervals being adjusted along the way - i.e. the momentary state of the channel is changing and the error message changes too.
If the other issue in googleads/google-ads-dotnet#59, I've unfortunately been unable to repro "name resolution" too. I agree with your findings that this is specific to the original reporter's environment.

Also please note:

There are fixes to the c-ares resolver in flight:

In the upcoming 1.20 release of gRPC (release planned next week), the DNS resolver library will be upgraded to a new version, which includes bug fixes. One can check if this will fix their problem by testing with the pre-release package (the 1.20.pre3 C# package is currently available on nuget.org).
https://github.com/grpc/grpc/pull/17688 has been found to fix some DNS resolution flakiness, this hasn't yet merged, but is in flight.

There has been one other issues related to use of the c-ares resolver on Windows that also hasn't yet been resolved. Plan to figure out how to address these in the next few weeks, stay tuned.

A few somewhat related points:

What does the grpc dns resolver do that a native dns resolver doesn’t? Is it a critical infrastructure part of grpc, or is it just a better dns resolver?

I am inclined to make this a flag in the Ads library with a default of true. I’d rather that my users don’t have to hunt for an Ads library fix on grpc repo. The experience is analogous to me sending a user to Microsoft to get a fix in the SDK. It works, but it is a terrible experience.

Ads users usually work with multiple ad networks, so bumping a library dependency on Ads library (e.g. Grpc.Core) will typically force them to retest their entire stack of integrations, which is something we want to minimize. Our current plans are to restrict any dependency updates to 3 times a year, aligned with our major version releases. Does grpc have stable milestones concept so we can pin to them?

@apolcyn Can you respond to the questions and resolve the issue?

Sorry for delayed response.

What does the grpc dns resolver do that a native dns resolver doesn’t? Is it a critical infrastructure part of grpc, or is it just a better dns resolver?

The c-ares resolver enables a couple of optional features, and has some behavior benefits:

the c-ares resolver enables grpclb-based load balancing to be bootstrapped by DNS - https://github.com/grpc/proposal/blob/master/A5-grpclb-in-dns.md for some background
the c-ares resolver enables "grpc service configs" to be resolved via DNS - https://github.com/grpc/proposal/blob/master/A2-service-configs-in-dns.md for some background

The c-ares resolver enables these features because these features rely on the ability to resolve "SRV" and "TXT" type DNS records. Now, there's a good chance that most users and/or services won't actually use these features (the grpc endpoints for common google cloud services don't currently use these features). However, because the c-ares resolver is enabled or disabled globally, I must warn that there might perhaps be issues if a user of the Ads SDK has e.g. their own grpc client connecting to their own grpc service which relies on those features, within the same address space. Also, should more common grpc service endpoints start using those features, we may want to revisit priority.
Just to throw it out, a "side benefit" of the c-ares resolver is that it's async/non-blocking, and doesn't use up a thread while it's doing DNS resolution. That might not be affecting users of your SDK, though.

I am inclined to make this a flag in the Ads library with a default of true. I’d rather that my users don’t have to hunt for an Ads library fix on grpc repo. The experience is analogous to me sending a user to Microsoft to get a fix in the SDK. It works, but it is a terrible experience.

I wouldn't personally object to using the native resolver by default. See caveats, things to be weary of in my above response, though.

Ads users usually work with multiple ad networks, so bumping a library dependency on Ads library (e.g. Grpc.Core) will typically force them to retest their entire stack of integrations, which is something we want to minimize. Our current plans are to restrict any dependency updates to 3 times a year, aligned with our major version releases. Does grpc have stable milestones concept so we can pin to them?

Can you elaborate on what you mean by "stable milestones"? gRPC has a six week long release cycle - every 6 weeks, we cut a new minor release. Bug fixes may get backported to those minor release as needed, depending on priority/risk.

Also note that the c-ares resolver code on Windows has had a few significant bug fixes since last comments, in particular

The c-ares library was upgraded as mentioned above
https://github.com/grpc/grpc/pull/17688
https://github.com/grpc/grpc/pull/19397

All of which have since been released (latest C# gRPC release being 2.23).


Whats the format of query in search follower?

How do you format a query - for example to say is this user following a certain user?
search_followers(user_id: int, query: str)

Look at it:
cl.user_friendship_v1
cl.user_friendships_v1

You can also select all followers by sorting through the alphabet:
import string

for letter in string.ascii_letters:
    cl.search_followers(user_id, letter)


Update Pages “pestinspectionsPage”

Automatically generated by Netlify CMS

👷 Deploy Preview for chic-shortbread-66a66b processing.

Name
Link

🔨 Latest commit
f5788208655bb3094d335406d203908df70ef6ba

🔍 Latest deploy log
https://app.netlify.com/sites/chic-shortbread-66a66b/deploys/6551fe1fcf86600008164aa6


Make the CLI more inclusive

Analyze and implement if necesary recommendations from: https://seirdy.one/posts/2022/06/10/cli-best-practices/

After review and discussion with the team, the actions to be taken are:

Improve the usage message when executing the --help command. Add usage examples.
Add --version option in the safe-cli and safe-creator commands to improve debugging.
Add message to be more user friendly once the safe information is listed.
Add a help subcommand to list all options inside a safe.
Add an exit subcommand to end the execution of safe-cli.


PR: Fix debugger with IPython 7.24+

Description of Changes
We need to avoid skipping hidden frames for that version because our debugger never stops on a frame.
Affirmation
By submitting this Pull Request or typing my (user)name below,
I affirm the Developer Certificate of Origin
with respect to all commits and content included in this PR,
and understand I am releasing the same under Spyder's MIT (Expat) license.

I certify the above statement is true and correct: @ccordoba12

Things need to be fixed in IPython, not here.


want to get the version of tesseract-ocr

when i run this project on my windows computer,  i got a result that be diffrent from to your site(https://ocr-example.herokuapp.com/), i want to find the reason, maybe is tesseract-ocr version or traindata

ok, it's because of the different versions.


Uninstalling blenderseed fails when attemping to delete appleseed.dll

I always need to do it in two steps:

Uninstall blenderseed -> leads to the error above
Restart Blender, uninstall again -> uninstalls correctly

The problem still exists. Not sure we can do anything about it.


Implement microformats test suite

@barnabywalters mentioned php-mf2 should run tests against the microformats test suite: https://github.com/tobiastom/tests. I have volunteered to take on this task.

It looks like the original test suite (https://github.com/microformats/tests) was updated so that this library can use it directly, instead of using the forked tobiastom version. Note the comment here: https://github.com/tobiastom/tests/issues/2#issuecomment-108883795
I would suggest we reference the main microformats/tests repo instead, ideally by publishing that repo to composer and bringing it in to here that way.

I've got this working on my fork of php-mf2. I have a PR open to add composer.json to microformats/test so it can be included here. Once that's accepted, I'll submit the test-suite PR for php-mf2.

Completed with #95.


Would be nice to expose a public variable to know whether or not a monitor view has been created.

For cases where you don't want to call start monitoring while there is already a monitor view active.
Or at least expose a read only performanceView property...

Actually after looking at startOrResumeMonitoring this shouldn't be an issue, nevermind.


Support for AWS NACLs and SGs

tasks list (by priority):

[x] #621
[x] #641
[x] #715
[x] #730
[ ] #812

closing
https://github.com/np-guard/vpc-network-config-analyzer/issues/812 is open


Can't set an initial default value

I'm using the latest version (2.2.4) and I can't set via html value or script a default value.
When jquery starts it hides the value and don't set the refered item.
Here is my code:
HTML
<input id="pickup" name="pickup" value="<?php echo $pickup ?>"
placeholder="<?php esc_html_e('Pick up location', 'transfers'); ?>"
type="text" class="origin-data-list flexdatalist form-control search-input" />

JS
var flexdatalistOptionsDefault = {
		minLength: 0,
		valueProperty: "id",
		visibleProperties: ["name", 'is_airport'],
		textProperty: "{name}, {country}",
		groupBy: ["country"],
		searchIn: "name",
		searchByWord: true,
		selectionRequired: false,
		noResultsText: "No results"
}

$(document).ready(function () {
	dataListInit(".origin-data-list");
});
function dataListInit(target) {
	var value = $(target).val();
	$(target).flexdatalist(flexdatalistOptionsDefault);
	if(value.length === 0 || !value.trim())
		$(target).flexdatalist('value',value)
}

Thank you very much in advance, good job!

Problem solved! The set value function of flexdatalist lib parse the value to string and doesn't find the id.


emoteadd Added.

Added emoteadd command that'll make things easier.

travis check tho. sigh always keep failing them ;-;


Try to offload Lipi Lekhika and exceljs to web worker for improved performance

also explore 3rd party tools which could help in this like partytown

font rendering in workers

Done in #58


Create assets

Based on my sketch, I want to create a quick drawing of some containers that I can use to hold everything together and make it look more finished.

I set up the clip file but don't have my stylus today. I'll whip up those drawing next time I can sit down.

This project is mainly just to practice the coding side. I will be making the assets as quickly as possible with just enough detail to get by. If I have time maybe I'll come back and make it pretty.


problem: identifier '__half_raw' is undefined

Hello! When I use chainer, there is a problem '...\cupy/carray.cuh(55): error: identifier '__half_raw' is undefined' . My IDE is anaconda spyder, win64, and the cupy version is 4.1. Please tell me how to fix this bug if it is convenient for you. Thanks!

Could you try using the latest (CuPy v4.4) Windows wheels (cupy-cuda80/90/91/92 etc.)? https://docs-cupy.chainer.org/en/latest/install.html#install-cupy (currently wheels are only available for Python 3.6 on Windows)
Could you share the output of set command? If you have multiple CUDA versions installed on your system, make sure to point the correct version in %PATH% environment variable.


Fix deprecated components and update to React v.16

This warning: warning.js:35 Warning: AutosizeInput: isMounted is deprecated. is caused by using a deprecated pattern, isMounted, instead of ComponentDidMount & ComponentWillUnmount.
Reference: https://facebook.github.io/react/blog/2015/12/16/ismounted-antipattern.html
This is mainly used in npm packages which are external libraries being used with React-dash.
Mainly the fixed-data-table , react-input-autosize & react-select packages in the node_modules dir. For these, we should probably file issues in their respective repos.
We are also using isMounted in a react-dash npm package, also in the node_modules folder. Looking at the files I think we may need to restructure the component to use the lifecycle methods (ComponentDidMount) etc. in order to fix this.

This ticket is no longer current; will create a new one if needed


Help: Unable to continuously read the data in the queue, and the response is invalid. My code is as follows:

Help: Unable to continuously read the data in the queue, and the response is invalid.
My code is as follows:
  // access to the event loop
    auto evbase = event_base_new();

    // handler for libevent
    AMQP::LibEventHandler handler(evbase);

    // make a connection
    AMQP::TcpConnection connection(&handler, AMQP::Address("amqp://localhost/"));

    // we need a channel too
    AMQP::TcpChannel channel(&connection);

    // create a temporary queue
    channel.declareQueue("hello").onSuccess([&connection, &channel](const std::string &name, uint32_t messagecount, uint32_t consumercount)
                                            {

        // report the name of the temporary queue
    std::cout << "declared queue " << name << std::endl;
     bool shouldExit = false;

    while (!shouldExit)
    {

        // consume messages from the "hello" queue
        channel.consume("hello").onSuccess([](const std::string &tag) {
            std::cout << "started consumer tag" << std::endl;
        })
        .onReceived([&channel](const AMQP::Message &message, uint64_t deliveryTag, bool redelivered) {

            std::cout << "received " << deliveryTag << std::endl;
            std::string data(message.body(), message.bodySize());
            std::cout << "Received: " << data << std::endl;

            // we remove the queue -- to see if this indeed causes the onCancelled method to be called
            // if (deliveryTag > 3) channel.removeQueue(queuename);

            // ack the message
            channel.ack(deliveryTag);

        });

        usleep(10000);

    }

        // run the event loop until there are no more events or timeout occurs

        // now we can close the connection
        connection.close(); });

    // run the loop
    event_base_dispatch(evbase);

    event_base_free(evbase);

in fact, when     while (!shouldExit) ,the code cannot get message anymore.but it can read a message when remove  the "while".
whatever,when read a message,  channel.ack(deliveryTag) can't ack to rabitmq.so ,I can  found the queue acked size is zero.
so,how to get correct result?

Remove the while loop, remove the sleep, just run the event loop using event_base_dispatch()


How to exclude a specific path and all the paths underneath from being cached?

/admin and all the paths underneath are not part of my app, so I want to avoid the SW from caching those paths.
I've tried with excludes: ['**/*.map', '/lighthouse/admin', '/lighthouse/admin/**'],, but this is not excluding /lighthouse/admin, nor /lighthouse/admin/whatever from being cached.
What am I doing wrong?

Full config:
new OfflinePlugin({
        safeToUseOptionalCaches: true,
        excludes: ['**/*.map', `/${appName}/admin`, `/${appName}/admin/**`],
        updateStrategy: 'changed',
        autoUpdate: 1000 * 60 * 2,
        caches: {
          main: [
            'assets/styles/app.*.css',
            'assets/scripts/vendor.*.js',
            'assets/scripts/app.*.js',
            'assets/scripts/manifest.*.js'
          ],
          additional: [
            ':externals:'
          ],
          optional: [
            ':rest:'
          ]
        },
        externals: [
          `/${appName}`
        ],
        ServiceWorker: {
          events: true,
          navigateFallbackURL: `/${appName}`
        },
        AppCache: {
          FALLBACK: {
            '/': '/assets/static/offline.html'
          }
        }
      })

Use cacheMaps option instead of navigateFallbackURL. The later is deprecated. With cacheMaps you can control whatever should be cached. Let me know if that works for you.

For AppCache, it most likely isn't possible. You cannot route separate routes to the network with it, but instead of have to route all your routes to offline. Unfortunately.

As a workaround for AppCache, you may do:
FALLBACK: {
  '/': '/assets/static/offline.html',
  '/admin': '/not-available.html'
}

In this case, when user is offline or the service is not available, it will show /not-available.html, i.e. custom error page.
@GGAlanSmithee do you think it would be good to document this on troubleshooting?

@NekR definitely. Will try to do that tomorrow! Is navigateFallbackURL marked as deprecated in the docs? Should I do it if not?

@GGAlanSmithee it's marked as deprecated in my mind and my plans, not in the docs just yet. It would be confusing to mark it as deprecated until I have appShell option released, which supposed to be as simple as navigateFallbackURL but use cacheMaps instead.
Once I have appShell done, we will update the docs about navigateFallbackURL :-)

appShell is actually done, but it's in 4.9.0 branch and I cannot merge it until other features in it are done..

@NekR could you please post here a cacheMaps config example for my use case? I think it would be useful also for other people with similar cases. Many thanks!

I've been trying with something like this, but still doesn't work :/
cacheMaps: [
          {
            match: /^(?!\/lighthouse\/admin).+$/,
            to: '/',
            requestTypes: ['navigate']
          }
        ],

It's better to use a function in this case. I'll show an example later.
On Oct 13, 2017 1:28 PM, "Joan Mira"<EMAIL_ADDRESS>wrote:

I've been trying with something like this, but still doesn't work :/
cacheMaps: [
{
match: /^(?!/lighthouse/admin).+$/,
to: '/',
requestTypes: ['navigate']
}
],
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/NekR/offline-plugin/issues/310#issuecomment-336414665,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABIlkcwrBkkzKB6_vKheMsZztAfXs951ks5srztegaJpZM4P3CuQ
.

ok, thanks. I'll wait for the example

cacheMaps: [
  {
    match: function(url) {
      if (url.pathname.indexOf('/admin/') === 0) {
        return;
      }

      return new URL('/', location);
    },
    requestTypes: ['navigate']
  }
]

Thanks, it works! :-)
Just had to slightly change it:
match: (url) => {
              if (url.pathname.indexOf('/admin/') === 0) {
                return true;
              }
              return new URL('/', url.location);
            },
            requestTypes: ['navigate']

Why it didn't work without true and why you use url.location which is undefined? URL construct should throw in your case .

I had to use url.location because location is not defined anywhere, so eslint complained. Also complained about returning nothing in one case and something on the other. I also had to add URL to the eslint globals. Did I do anything wrong?

Yes, you did. ESLint cannot know anything about your program or libraries especially so it shouldn't dictate what you do with them. It's perfectly fine to return whatever value in JS, that's why such functionality exists. Just disable ESlint for that function or use new Function(...) and do what I suggested you to do to fix the problem.

eslint

how you add URL to the eslint globals? i meet same question, Browser show: sw.js:3 Uncaught TypeError: Failed to construct 'URL': Invalid URL

eslint

how you add URL to the eslint globals? i meet same question, Browser show: sw.js:3 Uncaught TypeError: Failed to construct 'URL': Invalid URL

my problem solved, ignore me


Only initialize weld runtime once

Weld runtime is initialized when a weld module is compiled. This initialization overwrite the current runtime state. This is problematic when you try to run multiple different modules at the same time. This PR adds a guard to weld_runtime_init to make sure it is only initialized once.

cc @mateiz @sppalkia

Thanks!


Converting GGUF Models and Support for Smaller Models

I'm currently exploring the capabilities of the PowerInfer.GGUF framework and very impressed with results.
2 Questions:

I have some existing GGUF models and am interested in understanding the process for converting them into PowerInfer.GGUF models. Could you provide any guidance or documentation on how this can be achieved?

Support for Smaller Models: Also, I'm curious to know if there are plans to support smaller models in the future, particularly around the 3b size or even phi2. Such models can be incredibly useful for certain applications where resource constraints are a significant consideration (which is one PowerInfer can be even more relevant).

Thank you for your time and assistance.

These are indeed very important questions!
Converting Models: Currently, we support converting ReLU-based sparse models into the PowerInfer GGUF format.  For detailed information on model conversion, please refer to Model Weights. We use Python scripts for the conversion: /convert.py for the Llama family, and convert-hf-to-powerinfer-gguf.py for other architectures (only Falcon for now). Please note that direct conversion from GGUF to PowerInfer GGUF is not supported, as they are fundamentally incompatible.
Support for Smaller Models: We are very enthusiastic about expanding our support to smaller models, particularly those suitable for mobile devices. However, at present, we only support ReLU-based models, as outlined in our FAQs. We encourage you to stay tuned for updates on new model integrations!


First of all, thank you for the awesome plugin!
I noticed that syntax highlighting is lost if I set language attribute:

I can reproduce it on NeoVim 0.1.1

Hi! It's not really my plugin, the community contributed a lot :smile:
Use double quotes " instead of simple quotes in lang='stylus'. The regex is not checking for simple quotes (I don't think it's worth adding complexity to it)

I tried double quotes, but it doesn't work either:

@anvaka Are you sure the filetype is applied? :set ft=vue

I have the same issue


Is it possible to cancel an io device await?

For example, send a command via serial port; await its response but need to cancel it due to timeout.

Technically, it's possible to add a timeout argument that would cause the coroutine to "self-cancel". There's no way to cancel a coroutine from the outside, though if that's what you are asking.
QIODevice::read() doesn't accept a timeout argument and I'd like the QCoro API to stay as close to the Qt API as possible, but I could add a wrapper for QIODevice::waitForReadyRead(), so you would send the command and then do:
const bool ready = co_await qCoro(myDevice).waitForReadyRead(timeout);
if (ready) {
   const auto data = co_await qCoro(mDevice).readLine();
} else {
  // handle timeout
}

Would that work for you?

Thanks! This should work for me.
But I prefer if there is a common solution. Timeout is only one of the scenario. Another possible scenario is user click a button the cancel the command.
Is it possible that the API provides a context parameter so user can use the context to cancel the await?

You can't easily "cancel" a coroutine. In this case, coroutines are functions like any other, and you can't just cancel a running function. Even if the coroutine is currently suspended, destroying the associated state before the coroutine has finished would lead to memory leaks (at best).
Some operations are not cancellable, or cancellation is implementation defined, e.g. if you want to cancel a command you sent to a remote server do you just stop waiting for the response, or do you need to send something to the server to cancel processing the request etc.?
If you need cancellation for some of your operations, I'd suggest following:

you pass some kind of a cancellation token to your coroutine to allow cancelling the underlying operation
await the coroutine
if user clicks a button, you cancel the underlying operation using the cancellation token
when the awaited coroutine finishes, you check the state of the cancellation token to know whether the underlying operation has finished or whether it was canceled

you pass some kind of a cancellation token to your coroutine to allow cancelling the underlying operation

await the coroutine

if user clicks a button, you cancel the underlying operation using the cancellation token

when the awaited coroutine finishes, you check the state of the cancellation token to know whether the underlying operation has finished or whether it was canceled

This is exactly what I want. The scenario may be
// coroutine 1
bool ready = await co_await qCoro(myDevice).waitForReadyRead(main_cancellation_token);
if (!ready) {
     // some way to get the error status
} else {
    // handle IO
}

// main
void on_user_click()
{
    main_cancellation_taken.trigger();
}
void on_command_timeont()
{
    main_cancellation_token.trigger();
}

So, if the new API qCoro(myDevice).waitForReadyRead can accept a cancellation token, we can use it for both timeout and user trigger events.
Per my understanding, it is not easy for users of qCoro to provide the cancellation themself. We need some support from qCoro library

Sorry, I think cancellation tokens are out-of-scope for the current QCoro API, as I'm trying to stick as close to the original Qt API as possible (as QCoro is supposed to be just a wrapper for the Qt API). What I envisioned was that you would pass the cancellation token to your IO device, maybe by subclassing the Qt type you are using, and handle the cancellation of those operations yourself.
As a side-note, I've been prototyping a QCoroIO library with the idea of creating a fully asynchronous IO library for Qt (since I find the current Qt IO API rather frustrating). While I did not think of cancellation tokens initially, I think I'll include them in the API. I don't have any useful code to publish yet, though, so no promises :)

Thanks. Just go through the code deeper, I think I can create a derived a class from QCoroIODevice or write a new class base on it.


[bug] logout/login as different user does not update app state correctly

Description
If logged in to the SAT Frontend, then logout & login as a different user, the app does not refresh correctly for the new logged-in user.
Scenario/How to Reproduce

Log in as user A (here "Foo")
Log out
Log in as user B (here "Bar")

Expected Results
When logging in as a different user, the UI should reflect the current logged-in user
expect screen to look similar to this:

Actual Results
When logging in as a different user, part of the UI shows the correct logged-in user, and part of the UI is for the previous logged-in user.  (See image below: previous logged-in user was "Foo"; current logged-in user is "Bar")

Was resolved in a previous ticket.


internalizing i18n taglib (so old that cannot be found on maven

central). this will help remove the .tld and .jar from the coderepos of
all kie related repos.

http://github.com/droolsjbpm/kie-uberfire-extensions/commit/ccfa1ae34


Using codefences does not obey pygmentsUseClasses setting

Using Hugo v0.38.1
I've set the following in my config.toml and generated the styles.css file as per the docs.
If I use {{< highlight langcode >}}, the styles are used. If I use codefences, the styles are not applied.
This is because, when using codefences, the code is not wrapped in a div with the highlight style applied and the data-lang attribute is not added to the <code> tag.
This is certainly not the behaviour implied in the documentation and not the behaviour I would expect.
It makes migration from other platforms really difficult in addition to making it needlessly complex to write posts containing code.
Ideally, this would be fixed so that codefences have the same behaviour as implied in the current documentation. Otherwise, at least the documentation should be updated to reflect what actually happens.
As a footnote, it would also be helpful to noobs if the documentation about generating the syntax.css file were slightly expanded to explain that this file should go in the static folder and be included in the master layout baseof.html.
In case you think that I am being a little harsh, please let me also take this opportunity to say that Hugo is far and away the best static site generator that I've used. I've only just started using it and I've already been able to create a theme from scratch and migrate a rather gnarly WordPress site. So many thanks for the ongoing efforts.

If I use {{< highlight langcode >}}, the styles are used. If I use codefences, the styles are not applied.

We use both of these vairants on the Hugo docs site with pygmentsUseClasses, and that obviously works fine.

BTW, if you want examples, they are on my test site here:
https://confident-carson-5eac42.netlify.com/kb/hugo/hugo-snippets/

Make sure you set "pygmentscodefences = true" in your site config. I think you would be better off taking further troubleshooting at https://discourse.gohugo.io/

I already have that. I will take this across to discourse then.

For those interested in the Discourse post: https://discourse.gohugo.io/t/solved-using-codefences-does-not-obey-pygmentsuseclasses-setting/11442


Fix Wallet Standard event handler

From discussion with @ChewingGlass, if the Helium Wallet injection isn't ready by the time the app loads, the wallet won't get displayed. This is caused by improper handling of the wallet-standard:app-ready event, which is currently just logging the event. Instead, the handler needs to register the Standard Wallet.

Compare how the Wallet Standard's registerWallet function does this:
https://github.com/wallet-standard/wallet-standard/blob/8ac98a8ac6001a34906d01eac2ae0825f7208b32/packages/core/wallet/src/register.ts#L25-L39
You could use this function instead of the custom registration setup here.

Thankyou @jordansexton ❤

No problem @Perronef5! Please do test these changes out though -- I haven't tested them or even built the code.


_cameras = await availableCameras(); makes NSInvalidArguement Exception with iOS 17.

Steps to reproduce
Just run the sample project with iOS 17.3.1.
It will not get available cameras.
Expected results
All available cameras of the iOS device
Actual results
NSInvalidArgument Exception and App is terminating.
Code sample
Code sample
WidgetsFlutterBinding.ensureInitialized();
 _cameras = await availableCameras();

Screenshots or Video

Screenshots / Video demonstration
[Upload media here]

Logs
Logs
[Paste your logs here]

Flutter Doctor output
Doctor output
[✓] Flutter (Channel stable, 3.13.4, on macOS 14.3.1 23D60 darwin-x64, locale en-US)
    • Flutter version 3.13.4 on channel stable at /Users/fantasticdev/Documents/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 367f9ea16b (6 months ago), 2023-09-12 23:27:53 -0500
    • Engine revision 9064459a8b
    • Dart version 3.1.2
    • DevTools version 2.25.0

[!] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
    • Android SDK at /Users/fantasticdev/Library/Android/sdk
    • Platform android-34, build-tools 34.0.0
    • ANDROID_SDK_ROOT = /Users/fantasticdev/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b829.9-10027231)
    ! Some Android licenses not accepted. To resolve this, run: flutter doctor --android-licenses

[✓] Xcode - develop for iOS and macOS (Xcode 15.3)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 15E204a
    • CocoaPods version 1.13.0

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2022.3)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b829.9-10027231)

[✓] VS Code (version 1.87.0)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension can be installed from:
      🔨 https://marketplace.visualstudio.com/items?itemName=Dart-Code.flutter

[✓] Connected device (3 available)
    • sdk gphone x86 (mobile) • emulator-5554 • android-x86    • Android 11 (API 30) (emulator)
    • macOS (desktop)         • macos         • darwin-x64     • macOS 14.3.1 23D60 darwin-x64
    • Chrome (web)            • chrome        • web-javascript • Google Chrome 122.0.6261.112

[✓] Network resources
    • All expected network resources are available.

! Doctor found issues in 1 category.

Hi @fantasticDev2, do you have the necessary permissions in your Info.plist file? If so, can you share the full error that you get? I just tested the camera example on my device running iOS 17.3.1  and it worked as expected.


Confused on why this isn't working, confused about quantization.

I've been using 0cc4m GPTQ for a while and it's been smooth sailing, no errors. But when I try AutoGPTQ, it just, doesn't work. I'm trying to load Pygmalion 7b 4bit 32g, from TehVenom. And I get the following error:
Traceback (most recent call last):
  File "/home/XXX/Documents/Projects/Sapphire/main.py", line 63, in <module>
    main()
  File "/home/XXX/Documents/Projects/Sapphire/main.py", line 30, in main
    model = AutoGPTQForCausalLM.from_quantized(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/XXX/Documents/Projects/Sapphire/venv/lib/python3.11/site-packages/auto_gptq/modeling/auto.py", line 108, in from_quantized
    return quant_func(
           ^^^^^^^^^^^
  File "/home/XXX/Documents/Projects/Sapphire/venv/lib/python3.11/site-packages/auto_gptq/modeling/_base.py", line 791, in from_quantized
    raise FileNotFoundError(f"Could not find model in {model_name_or_path}")
FileNotFoundError: Could not find model in models/pygmalion-7b-4bit-32g

My file structure looks like this:
Project:
    models/
        pygmalion-7b-4bit-32g/:
            4bit-32g.safetensors
            config.json
            generation_config.json
            special_tokens_map.json
            tokenizer_config.json
            tokenizer.json
            tokenizer.model
    venv/
    main.py

My code is the following:
from transformers import (
    AutoTokenizer,
    pipeline,
    logging
)
from auto_gptq import (
    AutoGPTQForCausalLM,
    BaseQuantizeConfig
)

USERNAME = "Frantic"
VERSION = "1.0.0"
MODEL_DIR = "models/pygmalion-7b-4bit-32g"
MODEL_BASENAME = "Pygmalion-7b-4bit-GPTQ-Safetensors"
USE_TRITON = True

def main():
    print(f"Sapphire | Version {VERSION}")

    tokenizer = AutoTokenizer.from_pretrained(MODEL_DIR, use_fast=True)
    quantize_config = BaseQuantizeConfig(
        bits=4,
        group_size=32,
        desc_act=False
    )

    model = AutoGPTQForCausalLM.from_quantized(
        MODEL_DIR,
        use_safetensors=True,
        model_basename=MODEL_BASENAME,
        device="cuda:0",
        use_triton=USE_TRITON,
        quantize_config=quantize_config
    )

    logging.set_verbosity(logging.CRITICAL)

    prompt = "Hello, how are you today?"
    prompt_template = f"""### {USERNAME}: {prompt}
### Assistant:"""

    pipe = pipeline(
        "text-generation",
        model=model,
        tokenizer=tokenizer,
        max_new_tokens=512,
        temperature=0.9,
        top_p=0.9,
        repetition_penalty=1.15
    )

    print(pipe(prompt_template)[0]["generated_text"])

    input_ids = tokenizer(prompt_template, return_tensors="pt").input_ids.cuda()
    output = model.generate(inputs=input_ids, temperature=0.9, max_new_tokens=512)

    print(tokenizer.decode(output[0]))

if __name__ == "__main__":
    main()

Overall, I'm confused about what to do to fix it and understand AutoGPTQ. If anyone could help, that would be appreciated.

try changing your MODEL_BASENAME to 4bit-32g (the weights name without the .safetensors file extension)? sometimes these paths are fickle.

Hi @shakealeg, the model loading was improved in https://github.com/PanQiWei/AutoGPTQ/pull/383. Please pass the argument model_basename for custom model names, otherwise we greedily search quantize_config.model_file_base_name and models with basename f"gptq_model-{quantize_config.bits}bit-{quantize_config.group_size}g" or "model".
use_safetensors used to be False by default, which was turned to True by default in the above PR.
@CrazyBrick There is a WIP PR to support sharded models in AutoGPTQ: https://github.com/PanQiWei/AutoGPTQ/pull/364

Hi @shakealeg, the model loading was improved in #383. Please pass the argument model_basename for custom model names, otherwise we greedily search quantize_config.model_file_base_name and models with basename f"gptq_model-{quantize_config.bits}bit-{quantize_config.group_size}g" or "model".
use_safetensors used to be False by default, which was turned to True by default in the above PR.
@CrazyBrick There is a WIP PR to support sharded models in AutoGPTQ: #364

Thank you for you reply.I changed the code according to the PR#364(mainly in ``),but it dosen't work.
I print some variables while debugging:
`model_name_or_path:Qwen/Qwen-VL-Chat-Int4
isdir(model_name_or_path):True
model_save_name:Qwen/Qwen-VL-Chat-Int4/modelBut I haveQwen/Qwen-VL-Chat-Int4/model-00001-of-00005.safetensors Qwen/Qwen-VL-Chat-Int4/model-00001-of-00005.safetensors...(00001~00005) It won't find the whole model through one splitname(when I change the "model_file_base_name": "model"to"model_file_base_name": "model-00001-of-00005",then I will get NotImplementedError: Cannot copy out of meta tensor; no data!)
The following is the code I am using:
`from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
quantized_model_dir = "Qwen/Qwen-VL-Chat-Int4"
tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir, trust_remote_code=True)
use cuda device
model = AutoGPTQForCausalLM.from_quantized(
quantized_model_dir,
device_map="auto",
use_safetensors=True,
low_cpu_mem_usage=True,
trust_remote_code=True).eval()  # cuda
print(model.hf_device_map)
`
what should I do?

Hi @shakealeg, the model loading was improved in #383. Please pass the argument model_basename for custom model names, otherwise we greedily search quantize_config.model_file_base_name and models with basename f"gptq_model-{quantize_config.bits}bit-{quantize_config.group_size}g" or "model".
use_safetensors used to be False by default, which was turned to True by default in the above PR.
@CrazyBrick There is a WIP PR to support sharded models in AutoGPTQ: #364

Thank you for you reply.I changed the code according to the PR#364(mainly in ``),but it dosen't work.
I print some variables while debugging:
`model_name_or_path:Qwen/Qwen-VL-Chat-Int4
isdir(model_name_or_path):True
model_save_name:Qwen/Qwen-VL-Chat-Int4/modelBut I haveQwen/Qwen-VL-Chat-Int4/model-00001-of-00005.safetensors Qwen/Qwen-VL-Chat-Int4/model-00001-of-00005.safetensors...(00001~00005) It won't find the whole model through one splitname(when I change the "model_file_base_name": "model"to"model_file_base_name": "model-00001-of-00005",then I will get NotImplementedError: Cannot copy out of meta tensor; no data!)
The following is the code I am using:
`from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
quantized_model_dir = "Qwen/Qwen-VL-Chat-Int4"
tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir, trust_remote_code=True)
use cuda device
model = AutoGPTQForCausalLM.from_quantized(
quantized_model_dir,
device_map="auto",
use_safetensors=True,
low_cpu_mem_usage=True,
trust_remote_code=True).eval()  # cuda
print(model.hf_device_map)
`
what should I do?

Hi @shakealeg, the model loading was improved in #383. Please pass the argument model_basename for custom model names, otherwise we greedily search quantize_config.model_file_base_name and models with basename f"gptq_model-{quantize_config.bits}bit-{quantize_config.group_size}g" or "model".
use_safetensors used to be False by default, which was turned to True by default in the above PR.
@CrazyBrick There is a WIP PR to support sharded models in AutoGPTQ: #364

Hi, @fxmarty ,> Hi @shakealeg, the model loading was improved in #383. Please pass the argument model_basename for custom model names, otherwise we greedily search quantize_config.model_file_base_name and models with basename f"gptq_model-{quantize_config.bits}bit-{quantize_config.group_size}g" or "model".

use_safetensors used to be False by default, which was turned to True by default in the above PR.
@CrazyBrick There is a WIP PR to support sharded models in AutoGPTQ: #364

Thank you for you reply.I changed the code according to the PR#364(mainly in ``),but it dosen't work.
I print some variables while debugging:
model_name_or_path:Qwen/Qwen-VL-Chat-Int4
isdir(model_name_or_path):True
model_save_name:Qwen/Qwen-VL-Chat-Int4/model```
But I have `Qwen/Qwen-VL-Chat-Int4/model-00001-of-00005.safetensors` `Qwen/Qwen-VL-Chat-Int4/model-00001-of-00005.safetensors`...(00001~00005)
It won't find the whole model through one splitname(when I change the `"model_file_base_name": "model"` to `"model_file_base_name": "model-00001-of-00005",then I will get `NotImplementedError: Cannot copy out of meta tensor; no data!`)

The following is the code I am using:
from [Qwen-VL-Chat-Int4](https://huggingface.co/Qwen/Qwen-VL-Chat-Int4/tree/main)

from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
quantized_model_dir = "Qwen/Qwen-VL-Chat-Int4"
tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir, trust_remote_code=True)
use cuda device
model = AutoGPTQForCausalLM.from_quantized(
quantized_model_dir,
device_map="auto",
use_safetensors=True,
low_cpu_mem_usage=True,
trust_remote_code=True).eval()  # cuda
print(model.hf_device_map)
what should I do?

Hi @shakealeg, the model loading was improved in #383. Please pass the argument model_basename for custom model names, otherwise we greedily search quantize_config.model_file_base_name and models with basename f"gptq_model-{quantize_config.bits}bit-{quantize_config.group_size}g" or "model".
use_safetensors used to be False by default, which was turned to True by default in the above PR.
@CrazyBrick There is a WIP PR to support sharded models in AutoGPTQ: #364

Thank you for you reply.I changed the code according to the PR#364(mainly in ``),but it dosen't work.
I print some variables while debugging:
model_name_or_path:Qwen/Qwen-VL-Chat-Int4
isdir(model_name_or_path):True
model_save_name:Qwen/Qwen-VL-Chat-Int4/model

But I have Qwen/Qwen-VL-Chat-Int4/model-00001-of-00005.safetensors Qwen/Qwen-VL-Chat-Int4/model-00001-of-00005.safetensors(00001~00005)
It won't find the whole model through one splitname(when I change the "model_file_base_name": "model" to `"model_file_base_name": "model-00001-of-00005",
then I will get NotImplementedError: Cannot copy out of meta tensor; no data!
The following is the code I am using:
from [Qwen-VL-Chat-Int4](https://huggingface.co/Qwen/Qwen-VL-Chat-Int4/tree/main)
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
quantized_model_dir = "Qwen/Qwen-VL-Chat-Int4"
tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir, trust_remote_code=True)
# use cuda device
model = AutoGPTQForCausalLM.from_quantized(
    quantized_model_dir,
    device_map="auto",
    use_safetensors=True,
    low_cpu_mem_usage=True,
    trust_remote_code=True).eval()  # cuda
print(model.hf_device_map)

what should I do?

Hi @CrazyBrick, the model you are trying to use uses sharded checkpoints. This is unfortunately currently not supported in AutoGPTQ, there is a WIP PR open for it https://github.com/PanQiWei/AutoGPTQ/pull/364


Move to newest common scripts

Remove help.1 files as they are now created before the image build

No idea why the rhel tests is failing on pulling the base image... Going to try re-running tests

[test]

LGTM + tests are passing. So merging.
Thanks.


Fixed an issue that caused scroll to overshoot when using headers.

This fixes an issue where scrolling to a date that is at the end of the vertically-scrolling calendar caused overshooting.
Reproducing this issue requires the following:

Vertically-scrolling calendar that has enough content to make it scrollable
Calendar must use headers
Using .centeredVertically or .top for preferredScrollPosition when calling scroll
Scrolling to a date that is at the end of the calendar

This example is taken when scrolling to the last date of the calendar with preferredScrollPosition set to .top.
Before:

After:

maybe its not clear from the images, but is the 2nd image showing where it scrolled past the bounds? I do not see it in the image?

OK. I can re-create it when i change the itemSize to something smaller

Sorry, I should have just taken a GIF instead. :)
The Before image is where the calendar is scrolled to the last date with .centeredVertically. This causes the calendar to set the contentOffset of the scroll view past its contentSize. This is not a problem until you start to manually scroll the calendar, which causes a visual glitch as the contentOffset is instantly set to its maximum value.

Yea, its cool.
I recreated the issue. I have merged your fix


Drop support for Python 3.6, support Python 3.10

Closes #2912, closes #2913
Hello!
Pull request overview

Set python_requires in setup.py to >=3.7 instead of >=3.6.
Removed mention of supporting Python 3.6 in documentation.
Mention support for Python 3.10 in documentation.
Removed py36 tests in tox, and add py310 tests.

Note
I left mentions of Python 3.6 in the tools folder, e.g. in run_doctests.py:
https://github.com/nltk/nltk/blob/6411fabf05a75e2beb82b29fed614dbd12688f7f/tools/run_doctests.py#L10-L16
This folder seems mostly outdated and unused, also containing references back to jenkins and travis. One of the files, run_doctests.py, is not the proper way that doctests should be ran regardless. I didn't worry about these cases too much.
Thanks to @purificant for the commit of his I borrowed from his Python 3.10 branch :)
We should prepare a release soon.

Tom Aarsen

This PR is now planned to be merged after the upcoming NLTK release (3.6.6), allowing existing Python 3.6 users to benefit from our changes over the past months.

Merged, now that 3.6.6 is released.

Thanks @tomaarsen, @purificant

Weirdly enough - these tests passed, but every single CI run after this PR fails to build scikit-learn on Windows. I've removed CI tests for Windows 10 for now, until scikit-learn 1.0.2 releases (planned in middle December 2021), which supposedly adds support for Python 3.10 in Windows.


Memory checker (wmemcheck) completeness

This issue is meant to track what remains to make wmemcheck feature-complete and robust enough for practical use:

[ ] Handle SIMD loads and stores (#6829)
[ ] Hook calloc as well as existing malloc and free
[ ] Provide hooks for custom allocators to use
[ ] Allow for more flexibility in heap and stack layout (don't assume stack is above .data/.bss and everything below initial TOS is accessible)
[ ] Provide an alternate way to hook malloc/free, rather than requiring a name section in the .wasm module
[ ] Provide an option to print warnings as violations occur, rather than halting execution (like Valgrind)
[ ] Provide a mechanism to define "suppressions" for unwanted warnings (like Valgrind)
[ ] Track definedness at byte or bit level?

cc @iximeow for more thoughts, and others from today's Wasmtime meeting?

@alexcrichton noted earlier that it would be good to have:

[ ] an end-to-end test starting with some clearly-UB .c and test expectations for a stack trace pointing to wasmtime trapping a bad access

and since wmemcheck doesn't support multiple memories (though we ought to make it so!),

[ ] warn if a user tries to wmemcheck a multi-memory program
or add that support multiple memories. but i think that's maybe a step after "robust enough for practical use" in terms of effort, warning seems more expedient for now :)


splitting up the locator cache into a discovery and a batch locator cache

Issue
I've noticed an issue when ingesting metrics with new names into blueflood. The metrics always end up in Cassandra, but sometimes the metric names are not indexed in Elasticsearch.
Investigation
I've investigated the issue and it appears to be a problem with the LocatorCache and the fact that discovery and batch items are written in parallel threads.
In the HttpMetricsIngestionServer the discoveryWriter and batchWriter are applied right after each other on the metrics. When looking at such a DiscoveryWriter or BatchWriter, I noticed that they submit their work in a ThreadPool. So while it looks like one runs after the other, in fact they are running in parallel.
Locators are only marked current by one of the batch writers (AstyanaxWriter or DatastaxWriter), not by the discovery writer. However, both the DiscoveryWriter and the BatchWriter check whether a locator is current before writing it to their backend (Cassandra for the BatchWriter and ES for the DiscoveryWriter).
I think a false assumption was made here (or maybe the code evolved over time to threads) that the DiscoveryWriter runs before the BatchWriter, and therefore having the BatchWriter marking a locator as current is ok since the locator has already been written to ES. However, since this is not the case (they run in parallel), it's perfectly possible that:

the batch writer's thread runs slightly before the discovery writer's thread and writes the metric to Cassandra and marks the locator as current
then the discovery writer (writing the same metrics in parallel to ES) checks whether the locator is current, which it is since the batch writer just marked it as such.
because the locator is current, the discovery writer skips it, causing the issue of missing metrics in ES.

Solution
This PR tries to solve this issue by giving each writer its own LocatorCache. So I've created a static DiscoveryLocatorCache and a BatchLocatorCache, so that each writer has its own cache.
I've chosen this approach because it makes it easier to reason about the cache since it doesn't affect two different storage backends. That way you don't have to synchronise writes to ES and Cassandra, they can each handle the writes in their own thread and just update their own cache.
The DiscoveryWriter is now marking its own locators as current after having written them to ES, instead of relying on the batch writer.
This PR has been tested on our cluster and also includes a unit test for the DiscoveryWriter.

mmm, the failed integration test (HttpMultiRollupsQueryHandlerIntegrationTest) ran fine locally multiple times :(

The cache usually gets pretty big. By maintaining two(very similar) of these caches, we could be putting extra pressure on the JVM heap. I am wondering if there is a better way to handle this. May be decide whether the locator is current before sending it to either of these writers? Not sure if that would work. The solution you provided could be the last resort but I would like to try and see if there are other solutions.

@SlevinBE How about we do these changes in HttpMetricsIngestionServer? The below code does not compile but just to give you an idea. We basically remove any locators that are in cache before sending it to discoveryWriter. Once we make a call to discoveryWriter, it wont verify the cache anymore. This way, even when batchWriter runs slightly before discoveryWriter and marks the locator as current, we wont be affected.  What do you think?
            final List<IMetric> willIndex = DiscoveryWriter.condense(batches);

            discoveryWriter.apply(willIndex);
            return batchWriter.apply(batches);

While this would indeed solve the problem in the happy path, this however assumes that both the batchWriter and discoveryWriter are able to write the metrics successfully. In case the batchWriter does write the metrics successfully to Cassandra (and marks the locator as current) but the discoveryWriter encounters an error while writing to ES, then this would mean that all metrics with the same locator coming after this batch won't be provided to the discoveryWriter since they are marked as current.

Another option would be to keep a pair of Boolean values per locator entry in the LocatorCache, one indicating whether the discovery entry is current, the other indicating whether the batch entry is current.
Based on a simple test this would only increase the memory footprint for 1 million metrics by about 24MB (the current LocatorCache implementation uses about 240MB to hold 1 million metrics)
public class LocatorCache {

    private final Cache<String, LocatorCacheEntry> insertedLocators;

    public synchronized void setDiscoveryLocatorCurrent(Locator loc) {}
    public synchronized void setBatchLocatorCurrent(Locator loc) {}

    private class LocatorCacheEntry {
        private Boolean discoveryCurrent = false;
        private Boolean batchCurrent = false;

        ...
    }

}

SlevinBE Using the existing cache to keep a pair of boolean values sounds good to me.

Coverage increased (+0.1%) to 74.628% when pulling 79281022ef61e74c9ef2babb6fc815dd1e8e11ad on Klarrio:locator_cache_split into 2ef996a083892670f9118ba61e9ae55cfcc4bd01 on rackerlabs:master.

Coverage increased (+0.09%) to 74.575% when pulling 879b8c0947d06da3905e3f140867f670c6b49a02 on Klarrio:locator_cache_split into 2ef996a083892670f9118ba61e9ae55cfcc4bd01 on rackerlabs:master.

@ChandraAddala I've implemented the changes to the LocatorCache to keep a composite value of two booleans in the cache.

@ChandraAddala I've applied the requested changes.

Coverage increased (+0.08%) to 74.565% when pulling c25d8dbfa979d4b3123b5e9dea8f2189ede9406a on Klarrio:locator_cache_split into 2ef996a083892670f9118ba61e9ae55cfcc4bd01 on rackerlabs:master.