feat: add datetime tool

This tool empowers agents with the ability to determine the present date and time.
Its usefulness lies in liberating models from the constraints of being stuck in a snapshot of the past, where they were originally trained.

Thank you for the contribution - IMO this is a bit too trivial to do with a DynamicTool and specific with its return format.


FakeFileWrapper doesn't inherit from io.IOBase

Regular file objects inherit from io.IOBase, but the fake variant doesn't. Here's a test case that fails if the fs fixture is used, but passes otherwise:
import pytest
import io, os

def test_file_mro(fs):
    try:
        with open('foo.txt', 'w') as file:
            assert isinstance(file, io.IOBase)
    finally:
        os.remove('foo.txt')

if __name__ == '__main__':
    pytest.main([__file__])

I'm working on a file-object related library, so I really need this to work correctly.

Do you need that it actually inherits, or that it implements the io.IOBase methods, e.g. behaves like one?

I do actually need it to inherit. It's a bit hard to explain, but I have a whole bunch of functions that rely on the file objects' MRO to work correctly. The functions work differently depending on whether the file object inherits from io.TextIOBase, io.BytesIO, etc. So I actually need the whole MRO to be correct.
Here's the MRO of a file opened in text mode (open(mode='r') or open(mode='w')):
[<class '_io.TextIOWrapper'>, <class '_io._TextIOBase'>, <class '_io._IOBase'>, <class 'object'>]

A file opened with open(mode='rb'):
[<class '_io.BufferedReader'>, <class '_io._BufferedIOBase'>, <class '_io._IOBase'>, <class 'object'>]

And open(mode='wb'):
[<class '_io.BufferedWriter'>, <class '_io._BufferedIOBase'>, <class '_io._IOBase'>, <class 'object'>]

Hm, I have thought about this a bit, and I'm afraid that this is not something we are able to do without a major restructuring. We aim at emulating the functionality (e.g. it shall be possible to call every method as on the original object), but not on inheritance compatibility, which is an implementation detail that may change. Generally, I do not rely on MRO checks, because it is common in Python to emulate functionality by implementing interfaces instead of inheriting from the same base class (there are the abc classes, but they serve as an interface, not a base class implementation).
@jmcgeheeiv - do you have any thoughts on this?

I think you're wrong about the MRO being an implementation detail. I found this in the open docs:

The type of file object returned by the open() function depends on the mode. When open() is used to open a file in a text mode ('w', 'r', 'wt', 'rt', etc.), it returns a subclass of io.TextIOBase (specifically io.TextIOWrapper). When used to open a file in a binary mode with buffering, the returned class is a subclass of io.BufferedIOBase. The exact class varies: in read binary mode, it returns an io.BufferedReader; in write binary and append binary modes, it returns an io.BufferedWriter, and in read/write mode, it returns an io.BufferedRandom. When buffering is disabled, the raw stream, a subclass of io.RawIOBase, io.FileIO, is returned.

Well, you are right about the docs, though that doesn't change the fact that this will not be an easy change. I have been struggling with different IO behavior in different OSes and with different Python versions, and restructuring the code respectively is something I'm not looking forward to... so I may think about this, but don't expect something soon - sorry.

No problem. I'll just use real files for my tests, then.

Just for reference - this is only relevant for Python 3.

I have played around a bit with this, and didn't find a way to make this really working without a major rewrite. I will add a respective note to the documentation (limitations section) and close this issue.

Running your tests that use the real file system inside a Docker container would give you  the ephemerality of pyfakefs plus the exact implementation details like MRO.
Free software is so refreshing--no penalty for recommending the competition.

I would call it alternative, not competition 😛
Actually, I had been thinking about documenting alternatives with pros and cons, after the limitations paragraph. Docker is certainly the mostly used alternative today, at least under Linux. Maybe I'll add this before the next release...
I added a Features paragraph to the documentation with the last commit, maybe you can have a look if we are missing something here (I thought it makes sense to document the features alongside with the limitations...)


[ubuntu 16.04] code resource usage gradually increasing and eventually crashing system

VSCode Version:
1.13.1
379d2efb5539b09112c793d3d9a413017d736f89
OS Version: Linux my-system-name 4.4.0-81-generic #104-Ubuntu SMP Wed Jun 14 08:17:06 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Steps to Reproduce:

Just opened code with my home folder.

Even after I closed the code window one process is still running and its memory usage keeps increasing

After sometime my system will be unresponsive.

Updating C/C++ dependencies...

Downloading package 'C/C++ language components (Linux)'

The above update is running as soon as I open code and my internet is slow one. Haven't seen this issue till now

Reproduces without extensions: Yes

I wonder if this issue is being caused by an installed extension. Can you try to run VS Code without extensions? From the command line (NOT the integrated terminal in Code), execute: code --disable-extensions and try your steps again to see if it reproduces. If you see it is an issue with the extension, please file it against the extension repository itself.

I tried running without extension(code --disable-extensions) and it was the same. I doubt it has something to do with slow internet connection and updating C/C++ dependencies because when I am in good speed connection this is not happening

Can you try to reproduce with our nightly insider builds? You can give our preview releases a try from: http://code.visualstudio.com/Download#insiders


[AMBARI-24262] - Hive Client Restart Fails When Using A Credential Store

What changes were proposed in this pull request?
Hive clients cannot restart when a credential store is enabled. This is because the credential store uses the wrong config directory variable.
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HIVE/<IP_ADDRESS>.0/package/scripts/hive_client.py", line 63, in
    HiveClient().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute
    method(env)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 953, in restart
    self.install(env)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/<IP_ADDRESS>.0/package/scripts/hive_client.py", line 35, in install
    self.configure(env)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/<IP_ADDRESS>.0/package/scripts/hive_client.py", line 43, in configure
    hive(name='client')
  File "/usr/lib/ambari-agent/lib/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/<IP_ADDRESS>.0/package/scripts/hive.py", line 75, in hive
    params.user_group
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/security_commons.py", line 55, in update_credential_provider_path
    content = StaticFile(src_provider_path)
  File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 120, in action_create
    raise Fail("Applying %s failed, parent directory %s doesn't exist" % (self.resource, dirname))
resource_management.core.exceptions.Fail: Applying File['/usr/hdp/current/hive-client/conf/conf.server/hive-site.jceks'] failed, parent directory /usr/hdp/current/hive-client/conf/conf.server doesn't exist

How was this patch tested?
Manually tested in an environment where the credential store is enabled.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/3010/
Test PASSed.


Migrate process registry to here

https://eodc.atlassian.net/browse/EODCAPI-107
From writing the usage docs, I gathered that this is the most logical place for this to live, it’s weird to have to import a different repo for a basic usage tutorial. Also, to_callable directly depends on this class. An added benefit is that with this there's a chance that openeo-processes-dask could not depend on openeo-pg-parser-networkx in the future.

Looks fine to me! Does this mean that you plan to remove the process_registry from https://github.com/Open-EO/openeo-processes-dask/blob/main/openeo_processes_dask/core.py ?

Looks fine to me! Does this mean that you plan to remove the process_registry from https://github.com/Open-EO/openeo-processes-dask/blob/main/openeo_processes_dask/core.py ?

Yes!


/sbin/mount.juicefs: do not crash when called with no opts

$ /sbin/mount.juicefs mount
2022/08/12 16:13:18.700930 juicefs[5941] <WARNING>: [/sbin/mount.juicefs mount] [main.go:95]
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
github.com/juicedata/juicefs/cmd.reorderOptions(0xc000926340, 0x0, 0x0, 0x0, 0x0, 0x0, 0x19)
        /data/timfeirg/ce/cmd/main.go:174 +0xfb9
github.com/juicedata/juicefs/cmd.Main(0xc0008dbde0, 0x2, 0x2, 0x0, 0x0)
        /data/timfeirg/ce/cmd/main.go:84 +0x2054
main.main()
        /data/timfeirg/ce/main.go:29 +0x49

I don't think test failure are caused by my changes, any chance that we ignore this check?


Mislabelled 'Local Dataset path' Field

This could be me getting confused but I believe that the field which should be labelled 'Local Dataset path' is labelled as 'Local Datasource path'. The reasons I believe this to be the case are:

The field in question appears when the 'Include Data set' checkbox is checked.
If I check both 'Include data set' and 'Include Data Source', I see two textbox fields labelled 'Local Datasource path', presumably because one is intended to be for the local Data Source path and one is intended to be for the local Dataset path but is mislabelled.

More broadly, the labelling in this extension really needs tidying up. It's completely inconsistent, and when the names of these fields are so similar anyway and there are lots of controls visible on the page, it can get pretty confusing. For example, the extension refers to both "Data Source" and "Datasource", and "Data Set and Dataset". For what it's worth, I believe that the correct terms are "Data Source" (two words) and "Dataset" (one word).

Thanks for the comment! We will review it and come back with a solution

I've cloned the repo and fixed the problems with the labels and help text (for the Task that I use, at least). Can you allow me to make a Pull Request?

@philipstratford You can submit the Pullrequest and we'll review it

Thanks, that's what I'm trying to do but it says I don't have permission to push changes to the repository. It's possible that I'm not doing it right, I don't use GitHub much.

@philipstratford Giving you direct access, is not possible. But take a look at the usermanual:
https://help.github.com/articles/about-pull-requests/

I'd missed the fact that I had to fork the repo first. Pull Request submitted!

Build succeeded and latest version with your changes is available.
Thanks for the contribution :)


DO-NOT-MERGE Prod-Migr-plan: etc hosts for content-stode migr

Here we are removing the /etc/hosts entries for the content-store and
teh draft-content-store in Carrenza Production. This is part of the
migration plan, so the publishing-api and the frontends in carrenza
production will begin communicating with the content-store and
draft-content-store in aws by resolving via DNS.
Pair: @th31nitiate<EMAIL_ADDRESS>& @ronocg<EMAIL_ADDRESS>

Closing this, it was completed elsewhere


Added exception handling for ImportError

The import path munging does not succeed in some environments and we get an ImportError in those cases. Added exception handling to generate an empty yaml file in those cases.

Can one of the admins verify this patch?

Can one of the admins verify this patch?


[Enhancement] create table command should support binlog properties

Search before asking

[X] I had searched in the issues and found no similar issues.

Description
binlog.enable
binlog.ttl_seconds
Solution
No response
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

i will try


Finish initial MIDI support

With the new Extempore REPL, we can now do MIDI from LFE.
However, native-LFE MIDI will need to wait until more work on #22 is done.


Replace restify with express (and bunyan with winston)

At the moment, Falcon is stuck with restify@4 because restify@5 deprecated uncaughtException events. The latest version is restify@7.
Another issue that I've just come across is that restify pulls the logging module bunyan and the native module dtrace-provider as dependencies.
I'm opening this issue to get everyone's thoughts on replacing restify with express (and bunyan with winston). Please, could you add your :+1: / :-1: here?
@chriddyp @jackparmer @tarzzz @shannonlal @kndungu

I was using restify because it seemed "lighterweight" than express - we didn't need many of express's features like server-side templating, etc.
But from here on out, no preference from me, I'm good either way


Full drunc image for controller UI development

Description
Expands the docker image used by the drunc service to provide a fully functioned version of drunc capable of booting realistic test sessions. This is only required for development of the controller UI so we should decide whether we're doing that in this repository or as a separate app before merging.
Fixes # (issue)
Type of change

[ ] Documentation (non-breaking change that adds or improves the documentation)
[X] New feature (non-breaking change which adds functionality)
[ ] Optimization (non-breaking, back-end change that speeds up the code)
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] Breaking change (whatever its nature)

Key checklist

[ ] All tests pass (eg. python -m pytest)
[ ] The documentation builds and looks OK (eg. python -m sphinx -b html docs docs/build)
[ ] Pre-commit hooks run successfully (eg. pre-commit run --all-files)

Further checks

[ ] Code is commented, particularly in hard-to-understand areas
[ ] Tests added or an issue has been opened to tackle that in the future. (Indicate issue here: # (issue))

Is there any way that we could give devs the choice whether to use the full/lite images? E.g. by providing arguments to a helper script?

Definitely could. Worth doing if we decide to keep everything in one repo.

Looks good now :+1:

If someone could double check the latest merge commit that would be great. I think it's fine but may have missed something when the conflicts got a bit weird specifically in the README.
If all is okay then I can merge this in to #147

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 80.55%. Comparing base (90ab208) to head (870aa9c).

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #81   +/-   ##
=======================================
  Coverage   80.55%   80.55%
=======================================
  Files          26       26
  Lines         324      324
=======================================
  Hits          261      261
  Misses         63       63

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

LGTM @TinyMarsh

OK, with the help of @TinyMarsh I believe this is now ready for full review. I'll re-request from poeple who've already done one and pinging @AdrianDAlessandro for visibility.

Having some weird behaviours:
docker compose --profile drunc up

ends in
Error response from daemon: network 2892bd9abdea2fdd8cff6e493db78d55d98a11a8825874ccdc894ca1ee03344c not found

Also there is no indication that drunc or drunc-lite are even selected when running, or even if neither at all are chosen! Running without either seems to run fine. :shrug:

@jamesturner246 I don't see that error, but I am getting this one (Kafka related?):
kafka_consumer-1  | Listening for messages from Kafka.
kafka_consumer-1  | Traceback (most recent call last):
kafka_consumer-1  |   File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 105, in _execute
kafka_consumer-1  |     return self.cursor.execute(sql, params)
kafka_consumer-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kafka_consumer-1  |   File "/usr/local/lib/python3.11/site-packages/django/db/backends/sqlite3/base.py", line 354, in execute
kafka_consumer-1  |     return super().execute(query, params)
kafka_consumer-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

(etc. etc.)
I also tried just running docker compose up without setting --profile to see what would happen and it ran the full image. Maybe it defaults to whatever profile you selected last?

Going to assume you closed this by accident @alexdewar.

@alexdewar @jamesturner246 docker compose can sometimes get into a funny state if not all resources have been torn down previously and changes are made to the configuration. Could you both try a docker compose down -v --remove-orphans and take another pass.
The expected behaviour should be that:

running with --profile should run the relavant version of the drunc service
running without --profile should run all other services. In this case the app service will still start successfully and all views that don't involve communicating with the process manager will work. Opening the process manager UI will load the page but the table and message feed will not be populated (improvements to this planned in #150).

Going to assume you closed this by accident @alexdewar.

Yes, sorry!

With this merged you will need to update your local development workflow to either set the COMPOSE_PROFILES environment variable or use the --profile flag with docker compose.


Feature suggestion: option to include non-required properties in UI but exclude when empty from resulting JSON

We find the UI for including or excluding non-required properties to be a bit clunky and non-intuitive.  We'd prefer to just have all the non-required fields in the UI available to be edited, but have them get excluded from the constructed JSON unless data is entered.
Obviously, we lose the ability to distinguish between "property undefined" and "property value is empty string".  We don't mind that.
Here's a bit of code we're using that, in our case, has the desired effect.
var oldObjectEditor = JSONEditor.defaults.editors.object;
JSONEditor.defaults.editors.object = oldObjectEditor.extend({
    isRequired: function (editor) {
        // treat everything as required so that all fields show up in the UI
        return true;
    },
    refreshValue: function () {
        this._super();
        var required = this.schema.required;
        var hasRequired = $.isArray(required);
        for (var i in this.editors) {
            if (!this.editors.hasOwnProperty(i)) continue;
            if (hasRequired && required.indexOf(i) > -1) continue;
            var val = this.value[i];
            if (!val) delete this.value[i];
            else if ($.isEmptyObject(val)) delete this.value[i];
            else if ($.isArray(val) && val.length == 0) delete this.value[i];
        }
    }
});
JSONEditor.defaults.editors.object.options = oldObjectEditor.options;

We think this might be useful to other users and so propose that it be available as an editor option.

Thank you for this feature, it is very helpful. But I found a couple of corner cases. The option remove_empty_properties is not aware of the schema. It will be nice if it preserves the keys with empty values for the required fields.
For example, if the user enters nothing for following schema, the JSON will be
{
    "header": ""
}

because "header" is required and "innerObject" is not and all of its fields are empty, so it can be treated as an empty field itself. At the same time, if a user changes one field of the inner object, both fields appear
{
    "header": "",
    "innerObject": {
        "field1": "user input",
        "field2": ""
    }
}

which will make the JSON conforming to the schema with minimum of the user interaction
{
    "type": "object",
    "properties": {
        "required": [
            "header"
        ],
        "header": {
            "type": "string"
        },
        "innerObject": {
            "type": "object",
            "required": [
                "field1",
                "field2"
            ],
            "properties": [
                "field1": {
                    "type": "string"
                },
                "field2": {
                    "type": "string"
                }
            ]
        }
    }
}


Analyze fastpages forking

Forking a fastpages repo does not work in the sense, that the fork will not be able to run the blog by itself.
The main reasons are probably

Github actions don't have a deploy key that fits the fork
URLs may mismatch

In order to make a workflow possible where users can fork the blog in order to contribute it would be good to understand everything that is required to change a fork into a working copy of the blog. This would benefit users that would allow users to contribute without having to use local docker/docker-compose tools.
The goal of this issue is to collect actions necessary to make a fork working.
In another issue we'll then look into automation (github actions) and documentation to make the gathered steps easier for users.

It is probably helpful to look at the initial setup pull request #1. Which describes how to setup deploy keys, which probably needs to be done for the fork as well.
The optional section also describes the custom domain setup (which correlaid-codes makes use of) This might need to be reverted for the fork. In the end the forked blog should be reached under <github_username>.github.io/correlaid-codes. And we'll have to ensure that it doesn't just forward to the original blog.

I tried the following

Fork
Activate actions and add ssh keys according to #1
modified _config.yaml and changed username to my username, url to https://<github_username>.github.io, base_url to /correlaid_codes
changed base_url in settings.ini to /correlaid_codes
deleted the CNAME file

Steps 3-5 can be done in a single commit.
Afterwards the actions ran through without error and a copy of the blog was available under  https://<github_username>.github.io/correlaid-codes
The further ToDos that I see here are

Check that modification (styling/posts) of the fork update correctly on the blog
look into how a pull request would be done

Thoughts on pull request
In order to make a meaningful pull request (with and added blogpost for instance), the modifications that make the forked blog work, must not be part of the pull request. A way to achieve this might be the following (not tried yet)

Create a branch off the commit after forking and before the modifications described above
use git rebasing to commits with content changes (blog posts/styling etc.) to the branch without adding the commits that made the fork work. I'm not sure what the best git commands are git rebase -i would probably work well but con probably not be directly transferred to a github action.
create a pull request from the branch to the original repository


Regex matching

Alot of messages in NZ are in the same format (e.g. Ambo dispatch and Fire dispatch) so I use regex's to show data on them, e.g. fire messages all look like:
(PUKE431) 111-MIN 123 FAKE ST SAINT ANDREWS HAMILTON CITY. (XStr HEATH ST/BEERESCOURT RD) .ALARM ACTIVATION - RESIDENTIAL. #F1234567,
So I have a webpage that uses the regex info to for instance translate what "PUKE431" means etc.
Are you planning on implementing a similar feature?

Hmm, could be a really interesting feature. Can think of a few neat ways to do this... do you have an example of some of the regex you're using there? What would PUKE431 translate into?

In that example PUKE431 is the appliance so usually i put a thing in so you can hover over it and it says what ever the appliances full name is (e.g. Pukete 431).
This is the current Regex (please note it is in .net format) I use for NZ Fire Service:
((?.+?)) (?.+?)-(?.+?)( (Alarm Type (?.+?)))?( (Box (?.+?)))? (?.+?). (XStr (?.?)/(?.?))(? .)?(?.+?.)? (?(x-\d{1,10} y-\d{1,10}))?(?.*)? ?#(?F\d+)
Then i have a lookup list of all the values for each of the relevant field types. E.g. for FireCalltype i have a list item that says "STRU" is "Structure Fire" etc.

Cool, makes sense. Could expand the angular-highlight module to do something like that. It works on an array at the moment, applying HTML around keywords matching that array. E.g. the following is generated from having "ALERT" and "FIRECALL" as highlight keywords:

ALERT SpriHi - FIRECALL - #101722 - CARCOAR ST SPRING HILL

If we make it work with an array of objects you could have a keyword and a bit of "translation text" that shows up as a tooltip or something.

Sweet, that sounds good, I really want to switch to your system at some stage soon, as my one is getting a bit old and your one looks like it would be a good replacement.

One other suggestion is that over here we have two pager networks so maybe adding a network to the pagermessages table?

Easy :) Will call it a 'source' field, so it's flexible as either a name for the client machine/location or a network, etc.
Gotta do some DB restructuring in #30 anyway, so shouldn't be too big an issue.

Getting close to something workable for the regex matching - in your opinion would it be better to have the expanded text as a tooltip similar to how you have it now? Or would it be better to replace the text (maybe with original text in a tooltip)?
Demo: https://pagermon-hrng.c9users.io:8080/
So on that page I'm replacing "test" with "PUPPY", "fire" with "PARTY", and "[0-9][0-9][0-9][0-9]" with "numbers".

Also, with the source field - I'm thinking keep it to a real short identifier for optimal UX. Something like 4 or 5 chars max, would that work for you? Would be a column next to the capcode field, hidden on mobile.

re the Source - Yeah that would be great - currently I use really simple IDs for the network: Amb for Ambulance and TP2 for the main network.

re the Regex - i think putting tool tips would probably be the best as most of the time you know what it is but the time you do need to know what it means you can just hover over it...

Too easy, both sound good to me. :)

Ok cool, just testing the regex stuff but looks good so far:
https://pagermon-hrng.c9users.io:8080/
Settings:
http://i.imgur.com/D1GpJzO.jpeg

Sweet, that looks great! I will spin up a VM and test it out.

Awesome :) Have just merged the PR, so should be good to go.
Feel free to reopen if you run into any trouble!


Rebase to Horde 0.9.0

This rebases our changes to Horde 0.9.0 (the latest version).
Two additional changes:

This PR was reverted -- https://github.com/Whatnot-Inc/horde/commit/8fbaf33dab54db52cd373f6b707519dd69e4f86b
This change was incomplete and incorrect in the scenario when a process crashed on the same host. The restarted process might not have been re-added to the registry.
From horde's master - this PR was reverted - https://github.com/Whatnot-Inc/horde/commit/0d8c9aebdef20e1e4d3695c1ba2d4491d7bffae7
That's because we still care about the synchronicity (the PR mentioned in point 1) of the auction process start (that's what OTP's processes rely on), but it's not susceptible to the issue introduced in #2. We also don't care about the scenario that the PR fixed initially because we don't start two processes with the same key.

When this is approved, I will force push to master.

Just for my understanding:

This change was incomplete and incorrect in the scenario when a process crashed on the same host. The restarted process might not have been re-added to the registry

Why would the restarted process not be added to the registry if doing this synchronously @studzien?

Just for my understanding:

This change was incomplete and incorrect in the scenario when a process crashed on the same host. The restarted process might not have been re-added to the registry

Why would the restarted process not be added to the registry if doing this synchronously @studzien?

So I believe the problem was here, so essentially when another add operation was processed on the same node, it was ignored. I can't remember exactly, but I guess I wrongly assumed that handling this in register only would do the job.
I tested this by running this test:
https://github.com/Whatnot-Inc/whatnot_live/blob/main/test/medium/da/auction_test.exs#L408-L436
with and without sleep.


feat/bbox_scaling_parameter

Is your feature request related to a problem? Please describe.
Sometimes when partitioning a pdf with plots and tables, the plot title is being cropped off by the bounding box, which leads to you losing important context for summarization LLM.
Describe the solution you'd like
Add a bbox scale parameter to the partition function to increase/decrease the bounding box size.
Describe alternatives you've considered
Don't know of any alternatives, other than maybe changing the detection model.
Additional context
The change should be done in file: unstructured\partition\pdf_image\pdf_image_utils.py at line 183 (ver. 0.14.6).
Example of the code that should fix the issue:
offset = 0.18 # Should be a parameter
padded_bbox = cast(
    Tuple[int, int, int, int], pad_bbox((x1*(1-offset), y1*(1-offset), x2*(1+offset), y2*(1+offset)), (h_padding, v_padding))
)

Hi @LesykDev, Can you please provide a PDF document illustrating this issue?

Sure, @christinestraub — https://liftoff.energy.gov/wp-content/uploads/2023/05/20230523-Pathways-to-Commercial-Liftoff-Clean-Hydrogen.pdf
It is clearly visible on graphs at pages 9, 18 and 96.
Python 3.11.0, my unstructured version is 0.12.2

@LesykDev, Did you try adjusting the environment variable EXTRACT_IMAGE_BLOCK_CROP_VERTICAL_PAD? e.g.
os.environ["EXTRACT_IMAGE_BLOCK_CROP_VERTICAL_PAD"] = "20"


feat(prql-compiler): add DIALECT_NAMES to show available dialect names

Expose DIALECT_NAMES so that downstream software can get serialize representations of the available dialect names.
Like this (from eitsupi/prqlr#77)
pub fn dialects() -> Vec<&'static str> {
    prql_compiler::sql::DIALECT_NAMES.to_vec()
}

I am wondering if it might be better to change the serialization expressions from ansi to sql.ansi as well, since it is actually what is used in the PRQL script header.
@aljazerzen What do you think about renaming serialized expressions?
From my reading of the source code, it seems to me that the internal use is to remove the sql. so there seems to be no problem with adding sql. to the serialized expressions.

Looks reasonable to me.
I'm guessing this is because bindings would prefer strings to the Enum variants?
I don't have much experience on whether this is the best pattern to do that, but no objections for sure.

I'm guessing this is because bindings would prefer strings to the Enum variants?

Yes, this matches the target names that can be used in the headers of the actual PRQL scripts, so users should be able to easily find out what target names are available regardless of the version of prql-compiler they are using.

For example, we could use this to automatically update the following lines.
https://github.com/PRQL/prql/blob/e4bc3e2d83794142c027a526f848cf8a6326ce34/book/src/language-features/target.md?plain=1#L34-L44

I am wondering if it might be better to change the serialization expressions from name to sql.name (e.g. ansi -> sql.ansi) as well, since it is actually what is used in the PRQL script header.

I made this change in d11b3bceeebb3249a7208e751db5170bc372944f.
We can revert this or separate this into other PR.
I would like to hear your thoughts on this, thanks.

can we drop sql. prefix from the supported dialects?
They are all some form of sql and seem redundant.

can we drop sql. prefix from the supported dialect names?
They are all some form of sql and seem redundant.

This was recently done to set up a target for something other than SQL in the future.
https://github.com/PRQL/prql/blob/e4bc3e2d83794142c027a526f848cf8a6326ce34/CHANGELOG.md?plain=1#L189-L199

I still feel like maybe we drop sql.. Even if you add compile to new target system type, it will be the new platform name, like polars or something. I believe it's clear that the other dialects are databases and the compiled result will be sql.

I still feel like maybe we drop sql.. Even if you add compile to new target system type, it will be the new platform name, like polars or something. I believe it's clear that the other dialects are databases and the compiled result will be sql.

I empathize with this. I do think it has some benefits though — it's a hierarchy, it's easy to understand, it's extensible.
We also just made this change, so we should live with it for a while before we consider switching it again....


[BUG] -  Deprecated constants

Hi,

Would someone please fix these issues?
Thanks!

Sorry, duplication of previously opened.


SOLR-17158 Terminate distributed processing quickly when query limit is reached

In the case where partial results are not desirable, there is no reason to await any remaining requests, or to return any partial results. This change causes the &partialResults=false query parameter to short circuit processing and omit any results. This condition is indicated with a partialResults="omitted" value in the response (prior behavior was a value of true or absent)
https://issues.apache.org/jira/browse/SOLR-17158
This PR replaces https://github.com/apache/solr/pull/2379 which had become onerously stale.

looking back at https://github.com/apache/solr/pull/2493 I'm exploring if there are ways to also shorten the case where partial results are desired, but possibly due to super short CpuTimeAllowed in some tests, I'm occasionally (but not consistently) hitting cases where response builder hasn't done any work yet so various bits (such as requestIds) are null or otherwise uninitialized when we get to handleRequestBody. Exactly why is unclear, but some bits of responsebuilder maybe should be initialized by default. (i.e requestIds can be set to Collections.emptyMap() initially, since dozens of places don't check it for null)

Also, I've decided to presume that the failure to call transform in the ParallelHttpShardHandler was an oversight.

This is now feeling complete, I'll be looking to commit it tomorrow afternoon or perhaps Friday morning to main if nobody objects or requests more time to review.
I've had to back off any changes that mimic https://github.com/apache/solr/pull/2493
I had code for that, but I got lucky and hit a rare exception on one of my test runs, I found that ~1 in 20 times I was getting a CancellationException, and something less than 1 in 100 times (once it ran 800 iterations before failing) I was seeing a test fail where "omitted" was not properly reported with partialResults=false. I think I solved the CancellationException but ran out of time/energy to chase down whatever scenario was causing the rare test fail. Without that code (as the PR is now) I ran the CPU timeout test > 1500 times with no failures, so I'm confident that the problem is not transmitted in this PR, and additional enhancement can be done later. (I'll probably post a branch off of this branch showing what I had there for future reference).

As a sanity check I ran the jmh benchmarks on main and this PR branch. Here's the comparison:
https://docs.google.com/spreadsheets/d/1goRvpW1XEuLMrN_1BkB0OGHFYRDLUqo7E8XTCVvp9sU/edit?usp=sharing
Looks like most changes are noise, and the biggest changes are improvements I'm comfortable with this.

Ok I think this is ready, and I think I've answered the questions/feedback so far, so I'm putting it in main, but if issues arise or there's a strong feeling something needs to be reworked let me know, and we can fix it or revert if necessary. I want to move on to SOLR-17151. If things remain quiet until this ticket on Friday, I'll start looking at pushing a version of it to 9x


feat: Add Singapore UEN Recognizer

Change Description

Added SingaporeUENRecognizer class to the recognizers module.
Implemented logic for detecting and validating various UEN formats.
Added unit tests covering different UEN formats and scenarios

Issue reference
This PR fixes issue #1314
Checklist

[X] I have reviewed the contribution guidelines
[X] I have signed the CLA (if required)
[X] My code includes unit tests
[X] All unit tests and lint checks pass locally
[X] My PR contains documentation updates / additions if required

/azp run


Excessive enchanting player client crashes

Spark link
https://spark.lucko.me/RnthZWQksd
Expected behavior
The player client can enter normally and obtain items with high enchantments (greater than 255)
Observed/Actual behavior
When a player obtains or the client loads an enchanted item greater than 255, the connection will drop and they will not be able to re-enter
Steps/models to reproduce
Send an item with an enchanting level greater than 255 in the player's backpack
Purpur version

version
[22:31:22 INFO]: Checking version, please wait...
[22:31:25 INFO]: Current Purpur Version: 1.21.1-2304-dee41bc (MC: 1.21.1)*

You are running the latest version
Previous: 1.21.1-DEV-c5edeac (MC: 1.21.1)

Agreements

[X] I am running the latest version of Purpur available from https://purpurmc.org/downloads.
[X] I have searched for and ensured there isn't already an open issue regarding this.
[ ] I ticked all the boxes without actually reading them
[X] My version of Minecraft is supported by Purpur.

Other
purpur.yml
https://pastebin.com/S30TNHku
disconnect-2024-09-11_22.21.26-client.txt
https://pastebin.com/AtHp9Lpc

You have enabled unsafe enchants. This behavior is intended. Unsafe enchants hand you the wheel and responsibility for caring if the enchant will work on a level it was never intended to work with. You get to shoot yourself in the foot.


Build the govuk-elements-sass package to dist

This PR should be merged after the PR to replace Grunt with Gulp.

It moves the files needed by the govuk-elements-sass package into a temp directory, creates package.json file from the main package.json file used by the govuk-elements app and updates the npm ignore file to ignore all files, excluding the dist directory.
This will be a breaking change as the govuk-elements-sass package currently has a folder structure which includes /public/sass. This PR removes those two parent folders, so the package now contains:
README.md
CONTRIBUTING.md
LICENSE.txt
VERSION.txt
elements/*
_govuk-elements.scss
There is a further piece of work, after this to release this package to Github's releases using Travis.
What type of change is it?

Breaking change (fix or feature that would cause existing functionality to change)

Has the documentation been updated?

My change requires a change to the documentation.

Closing, I'll re-open when #459 has been merged, it'll make this much easier to review.


Fix menu icons mismatch

This PR fixes #95838 fixes #95864
Looks like icons were swapped and checkmarks were missing.
/cc @aeschli

Thanks @IllusionMH


[Feature] Autocomplete support for variables field when crafting GraphQL queries

GraphiQL already supports this, but having it inside Insomnia would be awesome.
How this works is, for example:
mutation NewEntityMutation($input: NewEntityInput!) {
	newEntity(input: $input) {
		entity { id name created updated description type }
	}
}

Where NewEntityInput is:
type NewEntityInput {
    id ID
    name String!
    description String
    type SomeTypeEnum!
}

When opening quotes in the variables bottom field, show a list to autocomplete the input variable itself, and inside, autocomplete to each of it's fields. It could even expand all the required fields at once (in this example, would insert the name and type keys). Since GraphQL is strongly typed, I guess autocompletion (to some extent) can be implemented for values as well.

Insomnia Version: 5.11.5
Operating System: Ubuntu 17.10

I completely agree with this, it would being incredibly helpful for expediting graphql calls. The other feature that I would love to see is an auto-fill feature for queries and mutations which would automatically enter all of the possible fields.
An example could be as follows:
mutation NewEntityMutation($input: NewEntityInput!) { 	newEntity(input: $input) { 		entity { ... } 	} }
If you enter '...' into then an auto-fill complete feature would pop up and you could automatically populate all of the fields.
I would be willing to try and take these on if they do not exist, or are not top priority.

@emil-nasso I was actually working on this this week.


dream.lock: injecting existing dependency

To fix packages that don't specify all their dependencies, it would be nice to have access to the dependencies so that you could do something like
dependencies: let
  getPkg = name: let version = elemAt 0 (attrNames dependencies.${name}); in { inherit name version; };
  missingDep = getPkg "foo";
  broken = getPkg "broken";
  brokenDeps = oldDeps.${broken.name}.${broken.version};
in
  dependencies // { ${broken.name}.${broken.version} = brokenDeps ++ [missingDep]; }

So then first the translator runs and then the dependencies can be patched before the cycle detection.
That way you can use the translator to determine the package details by adding the missing dep to the root dependencies and letting the translator resolve the impurity.
This could also be used to force using a certain version of a package etc.
For even more flexibility, the sources could also be passed into the function (or to a separate function) so they can be patched too.

It would also be great if this could be integrated with the overrides system. So then a known-broken package can always have the correct dependency injected.
Or maybe that is already possible?

Currently the way to inject dependencies manually is using the inject parameter as seen in the example of the main readme.
This is currently injected at evaluation time after translation. This will work with any translator and does not need to be supported by a translator.
Also for some translators, it is possible to add extra dependencies via argument. For example the pip translator allows specifying extraSetupDeps.
We do not have a way of adding deps via the override system so far. Maybe this could be integrated better. But I think first, we should have a proper module system in place before extending the current hacky API any more.
There were some discussions here: https://github.com/nix-community/dream2nix/issues/156
And I recently opened a PR to move functionality to nixos modules: https://github.com/nix-community/dream2nix/pull/249

Closing as injecting existing dependencies is already possible.
There are issues tracking improvements: https://github.com/nix-community/dream2nix/issues/361
If there are some other specific things we need to improve, let's open a specific issue for that


intel gpu: only enable xe driver starting with lunar lake

Description of changes
Things done

[x] Tested the changes in your own NixOS Configuration
[x] Tested the changes end-to-end by using your fork of nixos-hardware and
importing it via <nixos-hardware> or Flake input

cc @fidgetingbits

Changes look okay to me, although not tested it as on mobile atm.
My tiger lake asus laptop is using xe without explicitly using force probe, so not sure how that works if other models are requiring force probe.. maybe the whitelist mentioned in the other issue?
I don't plan to go back to i915 on my tiger lake laptop as I haven't had any issues with xe. Seems like it would be useful to have an option to still enable it on tiger lake.

Your chipset is probably in this whitelist: https://github.com/torvalds/linux/blob/ad46e8f95e931e113cb98253daf6d443ac244cde/drivers/gpu/drm/xe/xe_reg_whitelist.c#L26C37-L26C55
We can also override the driver again for your laptop model only as it uses the driver without any force option.

I added it back for zenbook.

@mergify queue


Update mixedrealityspatialdatapackager.md

Docs Build status updates of commit 708e481:
:warning: Validation status: warnings

File
Status
Preview URL
Details

mixed-reality-docs/mixedrealityspatialdatapackager.md
:warning:Warning
View
Details

mixed-reality-docs/mixedrealityspatialdatapackager.md

Line 4: [Warning] Invalid value for author: 'alfred-msft' is not a valid GitHub ID

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


metrics: fix incorrect unit of grpc source requests' duration.

What is changed and how it works?

Issue Number: ref #16265

What's Changed:
This pr is used to fix the incorrect format of the unit in the panel of grpc source requests' duration.
Before:

After correction:

Fix incorrect format unit of the `grpc source requests' duration` panel.

Related changes

[ ] PR to update pingcap/docs/pingcap/docs-cn:
[ ] Need to cherry-pick to the release branch

Check List
Tests

[ ] Unit test
[ ] Integration test
[ ] Manual test (add detailed scripts or steps below)
[x] No code

Side effects

[ ] Performance regression: Consumes more CPU
[ ] Performance regression: Consumes more Memory
[ ] Breaking backward compatibility

Release note

None

Did #17132 already fix this issue?

True, it's dup with https://github.com/tikv/tikv/pull/17132, which should be revert.


[jvm-packages] SIGSEGV Error on train() after using setHandleInvalid("keep")

When creating a pipeline object with stages that are:

StringIndexer
OneHotEncoder
VectorAssembler
XGBoostClassifier

When I apply setHandleInvalid("keep")  on the OneHotEncoders and StringIndexers, the process fails.
Sample executor log:
[16:17:22] WARNING: /xgboost/src/learner.cc:622: Tree method is automatically selected to be 'approx' for distributed training.
[16:17:22] Tree method is automatically selected to be 'approx' for distributed training.
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000000000093ad6, pid=13, tid=0x00007fefea019b10
#
# JRE version: OpenJDK Runtime Environment (8.0_212-b04) (build 1.8.0_212-b04)
# Java VM: OpenJDK 64-Bit Server VM (25.212-b04 mixed mode linux-amd64 compressed oops)
# Derivative: IcedTea 3.12.0
# Distribution: Custom build (Sat May  4 17:33:35 UTC 2019)
# Problematic frame:
# C  0x0000000000093ad6
#
# Core dump written. Default location: /opt/spark/work-dir/core or core.13
#
# An error report file with more information is saved as:
# /opt/spark/work-dir/hs_err_pid13.log
#
# If you would like to submit a bug report, please include
# instructions on how to reproduce the bug and visit:
#   https://icedtea.classpath.org/bugzilla
#

Driver Log
19/07/17 16:11:24 ERROR TaskSchedulerImpl: Lost executor 6 on <IP_ADDRESS>:
The executor with id 6 exited with exit code 134.
The API gave the following brief reason: null
The API gave the following message: null
The API gave the following container statuses:

The process succeeds when I change from keep to skip.

@ericpsimon Could you provide a reproducible script? And also OS, compiler you used if you compile it yourself, XGBoost version/commit.


Frontend crashes when entity contains FileRef attribute on entity form

For now we should ignore such attributes

There is now backend support for files.

Describe please which permissions user should has to have access to upload/download files

Environment:
Jmix version: 1.1.0
Jmix Studio plugin version: 1.1.SNAPSHOT3695-212
IntelliJ version: IntelliJ IDEA 2021.2.3 (Ultimate Edition)
AR:
unable to upload/download any file using frontend editor:

Test project
fr110.zip


Vala

I'm studying Vala to understand its potential.
Perhaps Tantalus might be written in Vala to follow elementary OS patterns, I'm not sure yet.

Bump!

I'll stick to Electron for now. I'm already studying so many things at the moment.

That would be so cool!!


Don't fail but print a warning if test target passed with -only option doesn't exist in the scheme.

We don't need this anymore.


Build site JSON parse error

Hello
I encountered the following problem while building Site.
$ hugo --minify
Building sites … ERROR 2020/06/09 20:56:12 JSON parse error: unexpected comma character on line 9 and column 20
    9:     "description": ,
                          ^
ERROR 2020/06/09 20:56:12 JSON parse error: unexpected comma character on line 9 and column 20
    9:     "description": ,
                          ^
Total in 8639 ms
Error: Error building site: failed to render pages: JSON parse error: unexpected comma character on line 9 and column 20
    9:     "description": ,
                          ^

But there are no errors when testing with other themes.
How can I fix it?
Thanks!

OS: Windows 10
Hugo: v0.72.0/extended
Hugo Theme: Zzo 4c77300

Later I discovered that it might be caused by content > en > publication.
Because I removed this folder and it works.

I have test with hugo commend and it worked.
I don't know why hugo --minify did not work.
I think it's the best choice to use hugo comment for building site now


Artifacts for v0.40 not available on Maven Central

Same problem as in #313 a while ago :)

Looks like it's fixed now, I'll re-open the ticket if the artifacts don't show up on Maven Central in a few hours.


Basic tests are failing on AWS EC2 instances with Contiv + Docker swarm

Netplugin Log :
https://gist.github.com/gaurav-dalvi/3337d0cdb22835ac4797d7c133c45283
Test Log:
https://gist.github.com/gaurav-dalvi/15191f29e9f6a3189dcb565ce2872c90

Based on: INFO[0181] docker: Error response from daemon: No healthy node available in the cluster., looks like something wrong with the setup itself.

will reopen when i see the issue.


Install rocpydecode on CI as sudo

Due to permission issue with installing rocpydecode on CI as extension
http://math-ci.amd.com/job/mainline/job/precheckin/job/rocPyDecode/job/develop/40/pipeline-console/?selected-node=543
http://math-ci.amd.com/job/mainline/job/precheckin/job/rocPyDecode/job/develop/40/pipeline-console/log?nodeId=568
building 'rocPyDecode' extension
creating build/temp.linux-x86_64-cpython-310/../../../../../../opt/rocm/share/rocdecode/utils
error: could not create 'build/temp.linux-x86_64-cpython-310/../../../../../../opt/rocm/share/rocdecode/utils': Permission denied

test run http://math-ci.amd.com/job/mainline/job/precheckin/job/rocPyDecode/view/change-requests/job/PR-111/6/pipeline-graph/

install as sudo to avoid permission error installing to /opt/rocm/lib and run as sudo to use system installed packages
http://math-ci.amd.com/job/mainline/job/precheckin/job/rocPyDecode/view/change-requests/job/PR-111/8/pipeline-graph/

http://math-ci.amd.com/job/mainline/job/precheckin/job/rocPyDecode/view/change-requests/job/PR-111/11/pipeline-graph/
passed on ubuntu, failed rhel/sles


[WIP] Added rbuilder to have reproducible builds

This PR is aimed to solve https://github.com/umee-network/umee/issues/9
Locally I could only run it successfully with linux/amd64. Will try to run it with linux/amd64 darwin/amd64 linux/arm64 darwin/arm64 when I have some more free time :)
The github workflow still needs to be updated, I don't have any experience with that, maybe some other contributor can jump in and avoid me lots of trial and error 😆 .

Thanks for the contribution @facundomedica! Happy to help with any workflow related issues. Also, there is an issue with macOS and Ledger https://github.com/tendermint/images/issues/70. I'd rather see that resolved first before we start using rbuilder.

@alexanderbez that's too bad! Seeing the response on the other issue, we might not want to use rbuilder at all? 😱

@facundomedica that's OK. I think we should still proceed with this PR. If we can have a method that produces reproducible hashes, that's good.

@alexanderbez should we keep the darwin platform?

@alexanderbez Also rbuilder doesn't support darwin/arm64. 😞

TARGET_PLATFORMS - whitespace-separated list of operating system/architecture pairs. linux/amd64, linux/arm64, darwin/amd64, and windows/amd64 are supported. Default: linux/amd64.

@facundomedica yes, we must support linux and macOS. I think we can skip darwin/arm64 for now until rbuilder supports it.

@alexanderbez Also rbuilder doesn't support darwin/arm64.

TARGET_PLATFORMS - whitespace-separated list of operating system/architecture pairs. linux/amd64, linux/arm64, darwin/amd64, and windows/amd64 are supported. Default: linux/amd64.

Hi there, rbuilder's author speaking here. Have you tried to build for darwin/arm64 anyway? When I left that message in the documentation probably darwin/arm64 was not yet officially supported.

@alessio I've tried building for MacOS but got an issue with Go's linker or something like that, I couldn't find a solution to it.
$ make build-reproducible
latest-build
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
--> Cleaning...
Building for darwin/arm64
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
go build -mod=readonly -tags "netgo ledger" -ldflags '-X github.com/cosmos/cosmos-sdk/version.Name=umee -X github.com/cosmos/cosmos-sdk/version.AppName=umeed -X github.com/cosmos/cosmos-sdk/version.Version=use-rbuilder-051f4b320c25ff7bcbfc4ad40143289d0b9acab7 -X github.com/cosmos/cosmos-sdk/version.Commit=051f4b320c25ff7bcbfc4ad40143289d0b9acab7 -X "github.com/cosmos/cosmos-sdk/version.BuildTags=netgo,ledger" -X github.com/tendermint/tendermint/version.TMCoreSemVer=v0.34.11' -o /tmp/tmp.TeQ5TnXyjZ/buildsources/build/ ./...
# github.com/umee-network/umee/cmd/umeed
/usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/tmp/go-link-892326816/go.o: file not recognized: file format not recognized
collect2: error: ld returned 1 exit status

make: *** [Makefile:71: build] Error 2
make: *** [Makefile:78: build-reproducible] Error 2

Thanks for the effort and contribution @facundomedica, but we're just going to stick with gorelease for our entire release pipeline, including reproducible build artifacts 👍

file not recognized: file format not recognized

Setting CGO_ENABLED=1 explicitly should make this go away


Proxy integration

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

Hey @triquanminh
I am investigating how companies are using superset after having forked the repo from Apache, I'd like to know how easy or difficult it is.
Would you be down to chat briefly on this topic?
Cheers,


Nuki Hub gets unavailable for a couple of hours a day - also tries to reconnect every minute afterwards

PROBLEM DESCRIPTION
Nuki Hub is several times a day unavailable via MQTT for several hours.

TO REPRODUCE
Let Nuki Hub run a few hours. Then it is shown as unavailable in HomeAssistant for a couple of hours. Later, it automatically reappears and works as designed.
EXPECTED BEHAVIOUR
Connection stays stable.
SCREENSHOTS
none
ADDITIONAL CONTEXT
Output from Mosquito broker:
2024-06-26 18:28:04: New client connected from <IP_ADDRESS>:60166 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:28:28: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:28:34: New connection from <IP_ADDRESS>:60167 on port 1883.
2024-06-26 18:28:34: New client connected from <IP_ADDRESS>:60167 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:28:58: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:29:04: New connection from <IP_ADDRESS>:60168 on port 1883.
2024-06-26 18:29:04: New client connected from <IP_ADDRESS>:60168 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:29:28: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:29:34: New connection from <IP_ADDRESS>:60169 on port 1883.
2024-06-26 18:29:34: New client connected from <IP_ADDRESS>:60169 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:29:58: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:30:04: New connection from <IP_ADDRESS>:60170 on port 1883.
2024-06-26 18:30:04: New client connected from <IP_ADDRESS>:60170 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:30:28: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:30:39: New connection from <IP_ADDRESS>:60171 on port 1883.
2024-06-26 18:30:39: New client connected from <IP_ADDRESS>:60171 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:31:04: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:31:10: New connection from <IP_ADDRESS>:60172 on port 1883.
2024-06-26 18:31:10: New client connected from <IP_ADDRESS>:60172 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:31:34: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:31:40: New connection from <IP_ADDRESS>:60173 on port 1883.
2024-06-26 18:31:40: New client connected from <IP_ADDRESS>:60173 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:32:04: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:32:13: Saving in-memory database to /data//mosquitto.db.
2024-06-26 18:32:15: New connection from <IP_ADDRESS>:60174 on port 1883.
2024-06-26 18:32:15: New client connected from <IP_ADDRESS>:60174 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:32:40: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:32:46: New connection from <IP_ADDRESS>:60175 on port 1883.
2024-06-26 18:32:46: New client connected from <IP_ADDRESS>:60175 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:33:10: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:33:16: New connection from <IP_ADDRESS>:60176 on port 1883.
2024-06-26 18:33:16: New client connected from <IP_ADDRESS>:60176 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:33:40: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:33:46: New connection from <IP_ADDRESS>:60177 on port 1883.
2024-06-26 18:33:46: New client connected from <IP_ADDRESS>:60177 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:34:10: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:34:16: New connection from <IP_ADDRESS>:60178 on port 1883.
2024-06-26 18:34:16: New client connected from <IP_ADDRESS>:60178 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:34:40: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:34:46: New connection from <IP_ADDRESS>:60179 on port 1883.
2024-06-26 18:34:46: New client connected from <IP_ADDRESS>:60179 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:35:10: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:35:29: New connection from <IP_ADDRESS>:60181 on port 1883.
2024-06-26 18:35:29: New client connected from <IP_ADDRESS>:60181 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:35:52: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:35:58: New connection from <IP_ADDRESS>:60182 on port 1883.
2024-06-26 18:35:58: New client connected from <IP_ADDRESS>:60182 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:36:22: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:36:28: New connection from <IP_ADDRESS>:60183 on port 1883.
2024-06-26 18:36:28: New client connected from <IP_ADDRESS>:60183 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:36:52: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:36:58: New connection from <IP_ADDRESS>:60184 on port 1883.
2024-06-26 18:36:58: New client connected from <IP_ADDRESS>:60184 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:37:22: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:37:36: New connection from <IP_ADDRESS>:60186 on port 1883.
2024-06-26 18:37:36: New client connected from <IP_ADDRESS>:60186 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:37:58: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:38:04: New connection from <IP_ADDRESS>:60187 on port 1883.
2024-06-26 18:38:04: New client connected from <IP_ADDRESS>:60187 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:38:28: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:38:39: New connection from <IP_ADDRESS>:60188 on port 1883.
2024-06-26 18:38:39: New client connected from <IP_ADDRESS>:60188 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:39:04: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:39:15: New connection from <IP_ADDRESS>:60189 on port 1883.
2024-06-26 18:39:15: New client connected from <IP_ADDRESS>:60189 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:39:40: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:39:51: New connection from <IP_ADDRESS>:60190 on port 1883.
2024-06-26 18:39:51: New client connected from <IP_ADDRESS>:60190 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:40:16: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:40:26: New connection from <IP_ADDRESS>:60191 on port 1883.
2024-06-26 18:40:26: New client connected from <IP_ADDRESS>:60191 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:40:52: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:41:01: New connection from <IP_ADDRESS>:60192 on port 1883.
2024-06-26 18:41:01: New client connected from <IP_ADDRESS>:60192 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:41:28: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:41:37: New connection from <IP_ADDRESS>:60193 on port 1883.
2024-06-26 18:41:37: New client connected from <IP_ADDRESS>:60193 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:42:04: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:42:12: New connection from <IP_ADDRESS>:60194 on port 1883.
2024-06-26 18:42:12: New client connected from <IP_ADDRESS>:60194 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:42:34: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:42:47: New connection from <IP_ADDRESS>:60195 on port 1883.
2024-06-26 18:42:47: New client connected from <IP_ADDRESS>:60195 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:43:10: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:43:23: New connection from <IP_ADDRESS>:60196 on port 1883.
2024-06-26 18:43:23: New client connected from <IP_ADDRESS>:60196 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:43:46: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:43:58: New connection from <IP_ADDRESS>:60197 on port 1883.
2024-06-26 18:43:58: New client connected from <IP_ADDRESS>:60197 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 18:44:22: Client nukihub has exceeded timeout, disconnecting.
2024-06-26 18:47:32: New connection from <IP_ADDRESS>:51014 on port 1883.
2024-06-26 18:47:32: New client connected from <IP_ADDRESS>:51014 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 19:02:14: Saving in-memory database to /data//mosquitto.db.
2024-06-26 19:32:15: Saving in-memory database to /data//mosquitto.db.
2024-06-26 20:02:16: Saving in-memory database to /data//mosquitto.db.
2024-06-26 20:13:26: New connection from <IP_ADDRESS>:54669 on port 1883.
2024-06-26 20:13:26: Client nukihub already connected, closing old connection.
2024-06-26 20:13:26: New client connected from <IP_ADDRESS>:54669 as nukihub (p2, c0, k15, u'mqttUser').
2024-06-26 20:15:23: New connection from <IP_ADDRESS>:59567 on port 1883.
2024-06-26 20:15:23: Client nukihub already connected, closing old connection.
2024-06-26 20:15:23: New client connected from <IP_ADDRESS>:59567 as nukihub (p2, c0, k15, u'mqttUser').

(Please, remember to close the issue when the problem has been addressed)

Same here. If mosquitto Broker output is needed please just tell.

I suggest trying the latest PR with Arduino Core 3 and IDF 5.1.
Besides many changes to the networking stack it also includes nuki hub specific improvements regarding MQTT.
You can find the binaries here: https://github.com/technyon/nuki_hub/actions/runs/9711394389
Do note that these binaries need to be flashed using USB/serial once because of changes to the partition table.

I tried flashing it with the Espressif Flash Download Tools, however it shows only 'null' in the command line and red fields in the UI

I find that to be a horrible tool and exclusively use esptools.
See https://docs.espressif.com/projects/esptool/en/latest/esp32/installation.html or https://www.flamingo-tech.nl/2021/03/21/installing-and-using-esptools/ for how to install esptools.
You can just use the command in the how-to-flash.txt file. You only need to change the COM port.

I did that now. However, the file seems to big for my ESP32:
Detecting chip type... ESP32
Chip is ESP32-D0WD-V3 (revision v3.1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz

esptool write_flash: error: argument  : Detected overlap at address: 0x8000 for file: nuki_hub.partitions.bin

Worked now. Different machine, different tool. I will update here how stable it is

Basically it works now. However, the main lock topic is unavailable

Is there a solution to unknown state?

I suggest trying the latest PR with Arduino Core 3 and IDF 5.1. Besides many changes to the networking stack it also includes nuki hub specific improvements regarding MQTT.
You can find the binaries here: https://github.com/technyon/nuki_hub/actions/runs/9711394389
Do note that these binaries need to be flashed using USB/serial once because of changes to the partition table.

with this binary, I cannot get a connection by ethernet anymore, specifically with a M5Stack Atom PoE Kit - W5500.
Nuki Hub does not respond anymore. If I disconnect it from the module, I can reach it by wifi.

I can conform that core 3 breaks both W5500 and LAN8720 ethernet support.

Lets keep this open until we have confirmation this is fixed in latest builds

I just flashed the latest build<PHONE_NUMBER>.1180.1), trying this now.

Works like a charm since then!

Thanks for reporting back.
Closing as completed

Unfortunately I'm still having issues with the lock being unavailable.
While the builds from main are better. They still haven't fixed it for me!

@adamlc Could you get serial logs while this is happening?

I'll see what I can do to capture them :)

Just a quick update. I've now had different behaviour without any changes.
MQTT is now reported as connected but updates aren't sent and everything seems out of sync with HA. Unfortunately I've not been able to get any logs at the time it happens.
Because of the instability I've had to sadly revert to 8.33 as that was rock solid before. All the new improvements to 8.35 did look fantastic and I hope with some more time I can debug.

Back on 8.33 which has since been rock solid!

And the problems you had with 8.35, or the 9.0 beta, or both?

Sorry for the late reply on this. Seems to now be resolved on the stable 9.0 builds 👍

It's really stable now with 9.0! Thanks for all the work 🍾


[Event Request] Table 6660 - Return Receipt Header - OnBeforePrintRecords

Hi AL-Team,
can you please add the following publisher to the procedure PrintRecords in table 6660 "Return Receipt Header"?
(This would then behave similar to table 110 - Sales Shipment Header)
procedure PrintRecords(ShowRequestForm: Boolean)
var
    ReportSelection: Record "Report Selections";
begin
    with ReturnRcptHeader do begin
        Copy(Rec);
>>>>>> NEW >>>>>>>>>>
        OnBeforePrintRecords(ReturnRcptHeader, ShowRequestForm, IsHandled);
        if IsHandled then
            exit;
<<<<<<<NEW<<<<<<<<<<
        ReportSelection.PrintWithDialogForCust(
          ReportSelection.Usage::"S.Ret.Rcpt.", ReturnRcptHeader, ShowRequestForm, FieldNo("Bill-to Customer No."));
    end;
end;

[IntegrationEvent(false, false)]
local procedure OnBeforePrintRecords(var ReturnRcptHeader: Record "Return Receipt Header"; ShowDialog: Boolean; var IsHandled: Boolean)
begin
end;

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.


Added Support for defining file types in fcontext defined type

Right on! Thanks for the code.

Awesome! I'll have to work with the fork jgazeley has up on puppet forge to get it included there (https://forge.puppetlabs.com/jgazeley/selinux )
--- Original Message ---
From: "James Fryman"<EMAIL_ADDRESS>Sent: January 12, 2015 10:50 AM
To: "jfryman/puppet-selinux"<EMAIL_ADDRESS>Cc: "exodusftw"<EMAIL_ADDRESS>Subject: Re: [puppet-selinux] Added Support for defining file types in fcontext defined type (#21)
Merged #21.

Reply to this email directly or view it on GitHub:
https://github.com/jfryman/puppet-selinux/pull/21#event-217501244

Let me just push this up to the forge.

https://forge.puppetlabs.com/jfryman/selinux


docker ps -a --filter 'exited=0' hangs, causes any further docker ps to hang

Per the title, if I do a docker ps -a --filter 'exited=0', it hangs and makes all future calls to docker ps hang:
$ service docker start
docker start/running, process 12984
$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
$ docker ps -a --filter 'exited=0'
^C$ docker ps
^C$

I have to restart the daemon to be able to do docker ps again.
Here's an strace.
This is all happening on docker 1.3.1.  I don't know if it's related to #7905 or #7999 so opened a new issue to be safe.
$ docker version
Client version: 1.3.1
Client API version: 1.15
Go version (client): go1.3.3
Git commit (client): 4e9bbfa
OS/Arch (client): linux/amd64
Server version: 1.3.1
Server API version: 1.15
Go version (server): go1.3.3
Git commit (server): 4e9bbfa

$ docker info
Containers: 62
Images: 656
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 781
Execution Driver: native-0.2
Kernel Version: 3.13.0-36-generic
Operating System: Ubuntu 14.04.1 LTS
WARNING: No swap limit support

edit: docker ps --filter 'exited=0' without the -a is okay and doesn't hang; it's when the flags are combined that causes the issue. Also I noticed I was mixing long and short flags, but docker ps -a -f 'exited=0' also hangs.

I am using docker version 1.6.2 and verified that this issue got fixed. Good job Docker folks.


handle github remote urls without .git suffix

Github#slug doesn't correctly parse remote urls if they don't end in .git even though cloning from github (at least, haven't tested any other servers) works happily with the same urls. if this is a universal thing i'm happy to make the same change for gitlab as well.

@modosc could you add an entry to CHANGELOG?

@modosc update CHANGELOG myself, no worries :smile:
Thanks :bow:


Source maps are only generated when not using PSC IDE fast rebuilds

When I enable fast rebuilds by adding pscIde: true to the options, the sourcemaps are not being updated.

How to fix this issue while still retaining the fast rebuild feature? I can't disable fast rebuild, as I have 1500 modules, and compiling is taking me 6 seconds to complete and dev-server auto reload to trigger.

The ability to do this was added to purs ide in 0.12.1: https://github.com/purescript/purescript/pull/3449
Check out the codegen parameter in here: https://github.com/purescript/purescript/blob/9446c8bb5f6cd5e74b3f1eb787cfded684391400/psc-ide/PROTOCOL.md#rebuild
You'll need to thread through the configuration parameter through purs-loader and then things should work with codegen: ["js", "sourcemaps"].

Glad to know that it is already solved, but unfortunately I'm still on 0.11.7 due to some internal libraries not being updated till now. Will try to update them first. Please tell if there is a solution for 0.11.7 users too.

Unfortunately not. In 0.11.7 land the purs ide rebuilds can't produce sourcemaps.

Good, now I have one more compelling reason to ask internally to update the libraries. Thanks for the response.

Thanks for the thread and replies. This might be good information to add to
the readme of the loader. Thanks!
On Thu, Dec 27, 2018 at 08:35 Sri Harsha Chilakapati <
<EMAIL_ADDRESS>wrote:

Good, now I have one more compelling reason to ask internally to update
the libraries. Thanks for the response.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ethul/purs-loader/issues/116#issuecomment-450151784,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAVYy--iJJIjiQrh2dGbcQmIcWpSesv9ks5u9MyjgaJpZM4Zioik
.

I don't think this is purely a documentation issue, the extra parameter will need to be added to every "rebuild" request sent to purs ide, so a tiny bit of configuration will need to be added to purs-loader and threaded through.


Create Product Card 3 black

One need to make this component using HTML and Tailwind CSS. I would suggest to use Tailwind Playgrounds to make things faster and quicker.
Here is a reference to the component.

Follow figma design and to make the button use Button Design System
After building the component please raise a PR with a screenshot of the component and add the component in Product Cards/src/product-card-3.html & rename the file to product-card-3-black.html
If you need to use any icon please use it from Hero Icons
Good luck.

i want to work on this issue please assign it to me under Jwoc

hey ! @RavindraP04 product-card-3-black.html already exist , so can i add as product-card-3-blue.html ?
https://user-images.githubusercontent.com/110401845/220354345-dbe52f2a-3eae-4752-b602-0a6f06c1a473.mp4


Delete Error?

I use like this:
var repo = _unitOfWork.GetRepository();
repo.Delete(id);
_unitOfWork.SaveChanges();
But it Error:
Microsoft.EntityFrameworkCore.DbUpdateConcurrencyException:“Database operation expected to affect 1 row(s) but actually affected 0 row(s). Data may have been modified or deleted since entities were loaded.
at Repository.cs line 548 code is ：
var entity = Activator.CreateInstance();
property.SetValue(entity, id);
_dbContext.Entry(entity).State = EntityState.Deleted;

First of all, ensure database has the record with id;
Secondly, ensure whether the table has combining primary keys;

First of all, ensure database has the record with id;
Secondly, ensure whether the table has combining primary keys;

Yes, the two is ok. I use Mysql. It still error.
When it runs code at Repository.cs line 548 ,It's Error
var entity = Activator.CreateInstance();
property.SetValue(entity, id);
_dbContext.Entry(entity).State = EntityState.Deleted;
When it runs code at Repository.cs line 554, It's OK
var entity = _dbSet.Find(id);
if (entity != null)
{
Delete(entity);
}

OK, I'll test this. Right now, using Delete(entity) for bypassing this issue.

@xyting ok, thanks!

any update?


twitter redirect biased to nitter.net

I did this 9 times and got this 8 times:
> curl -I -A iPad https://farside.link/twitter.com/mikko/status/1692503249595584526
HTTP/1.1 302 Found
location: https://nitter.net/mikko/status/1692503249595584526

and this once:
> curl -I -A iPad https://farside.link/twitter.com/mikko/status/1692503249595584526
HTTP/1.1 302 Found
location: https://nitter.it/mikko/status/1692503249595584526

at this point I might as well use nitter.net directly.

It's not biased, it just so happens that nitter.net is typically the first to roll out fixes to its scraping algorithm, and seems to have fewer issues than other instances. Although, running your same test didn't return nitter.net for me at any point. There isn't any code to prefer one instance over another though, it just selects a random one from a pool of working instances and then prefers that instance for a short period of time afterwards.


Native storybook with storyshots broken when using knobs

When using knobs addon in native storybook, storyshots fail
 Storyshots › Button › with some emoji

    TypeError: Cannot read property 'STORYBOOK_ENV' of undefined

I made a reproduction of the problem:
https://github.com/ktj/storyshots-knobs-repro
run yarn test
Problem line: https://github.com/storybooks/storybook/blob/master/addons/knobs/src/index.js#L74
window is undefined in native

Adding a script with global.window = global; to my Jest config's setupFiles array fixed this for me. Having window as a global fixes the undefined error which allows the default case to execute.


[misc] Optimize speculative decoding

This PR introduces optimizations to the speculative decoding process by reducing memory copy overhead, resulting in improved performance.
After profiling, I found that cuda memcpy occurred abnormally, so I improved the code. (Refer to the profiling image below)
I experimented with one a100 gpu, and the code I used for the experiment is as follows.
I improved the _filter_model_output function and _merge_outputs function, and confirmed that the test code below improved by about 3%.

Applied Function
E2E Latency

baseline
0.462s

_filter_model_output
0.457s

_merge_outputs
0.459s

_filter_model_output + _merge_outputs
0.449s

Test code using the OPT model
import time
import nvtx
from typing import List
from vllm import LLM, SamplingParams

def time_generation(llm: LLM, prompts: List[str],
                    sampling_params: SamplingParams):
    llm.generate(prompts, sampling_params)
    llm.generate(prompts, sampling_params)
    times = []
    with nvtx.annotate("generate", color="blue"):
        for _ in range(10):
            start = time.time()
            outputs = llm.generate(prompts, sampling_params)
            end = time.time()
            times.append(end-start)
    print(sum(times) / len(times))

    for output in outputs:
        generated_text = output.outputs[0].text
        print(f"text: {generated_text!r}")

if __name__ == "__main__":
    sampling_params = SamplingParams(temperature=0.0, max_tokens=100)

    prompts = [
        "The president of the United States is",
    ]

    llm = LLM(
        model="facebook/opt-6.7b",
        tensor_parallel_size=1,
        speculative_model="facebook/opt-125m",
        num_speculative_tokens=5,
        use_v2_block_manager=True,
    )

    time_generation(llm, prompts, sampling_params)

Profiling results

Before code improvement

After code improvement

Thanks @jacob-crux! It looks like this mostly comes down to indexing using an on-device tensor rather than a list. This can also be used with the existing tensor[index_tensor] syntax ... are the index_ tensor methods much faster?

Hello!
First, I confirmed that when slicing with output_indices_to_retain, which is a List[int] in the _filter_model_output function, cudaMemcpy is used to copy and sync the index every time.
When comparing the speed difference between the list slice kernel and torch.index_select with the test code, I found that there was a speed difference of about 40-50% when comparing only the kernel. However, since the proportion of the existing kernel is small, it is difficult to get a big feeling.
Below are the profiling results. The first result is from the existing code, the second result is when only the index is assigned to a CUDA tensor, and the last result is from the code using torch.index_select in the PR.

Thanks @jacob-crux, I wasn't questioning that using repeated list-based slicing is slower. I was suggesting that the same syntax can be  used to slice using a tensor. Then the actual code modifications needed would be minimal.
i.e.
index_list = [1, 3, 5]
index_tensor = torch.tensor(index_list, device=device)
sliced = a_tensor[index_tensor]
sliced2 = another_tensor[index_tensor]

Thanks @jacob-crux, I wasn't questioning that using repeated list-based slicing is slower. I was suggesting that the same syntax can be used to slice using a tensor. Then the actual code modifications needed would be minimal.
i.e.
index_list = [1, 3, 5]
index_tensor = torch.tensor(index_list, device=device)
sliced = a_tensor[index_tensor]
sliced2 = another_tensor[index_tensor]

@njhill Thank you for your review.
The comment was an experiment with the code you mentioned.
Here is a simple code example of the 3 profiled contents in the picture in the comment.
# profile 1
index_list = [1, 3, 5]
sliced = a_tensor[index_tensor]

# profile 2
index_list = [1, 3, 5]
index_tensor = torch.tensor(index_list, device=device)
sliced = a_tensor[index_tensor]

# profile 3
index_list = [1, 3, 5]
index_tensor = torch.tensor(index_list, device=device)
sliced = a_tensor.index_select(0, index_tensor)

The difference mentioned here is

The difference between profile 1 and profile 2 is that memcpy occurs frequently.
The difference between profile 1&profile 2 and profile 3 is the difference in the GPU kernel called when slicing, and as in the middle section of the picture, the kernel shows a speed difference of about 40~50%.

Although the code of profile 2 may look a little simpler, the kernel of profile 3 has the effect of improving performance on the GPU.


capnproto: 1.0.1 -> <IP_ADDRESS>

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://repology.org/project/capnproto/versions.
meta.description for capnproto is: Cap'n Proto cerealization protocol
meta.homepage for capnproto is: https://capnproto.org/
Updates performed

Version update

To inspect upstream changes

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found <IP_ADDRESS> with grep in /nix/store/ymy2yfvxy6xr3ng23c9xw8bgl5ixv0lj-capnproto-<IP_ADDRESS>
found <IP_ADDRESS> in filename of file in /nix/store/ymy2yfvxy6xr3ng23c9xw8bgl5ixv0lj-capnproto-<IP_ADDRESS>

Rebuild report (if merged into master) (click to expand)

23 total rebuild path(s)

23 package rebuild(s)

First fifty rebuilds by attrpath
brise
capnproto
capnproto-java
capnproto-rust
cargo-rr
conmon-rs
ekam
emacsPackages.rime
fcitx5-rime
flowgger
ibus-engines.rime
laminar
librime
python310Packages.pycapnp
python311Packages.pycapnp
rime-data
rr
sonic-lineup
sonic-visualiser
surelog
uhdm
veilid
yosys-synlig

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/ymy2yfvxy6xr3ng23c9xw8bgl5ixv0lj-capnproto-<IP_ADDRESS> \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A capnproto https://github.com/r-ryantm/nixpkgs/archive/6db5b06f027fae8e6d964fddb104bc97af0e3f23.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/6db5b06f027fae8e6d964fddb104bc97af0e3f23#capnproto

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/ymy2yfvxy6xr3ng23c9xw8bgl5ixv0lj-capnproto-<IP_ADDRESS>
ls -la /nix/store/ymy2yfvxy6xr3ng23c9xw8bgl5ixv0lj-capnproto-<IP_ADDRESS>/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

2 packages marked as broken and skipped:

python310Packages.pycapnp
python311Packages.pycapnp

21 packages built:

brise
capnproto
capnproto-java
capnproto-rust
cargo-rr
conmon-rs
ekam
emacsPackages.rime
fcitx5-rime
flowgger
ibus-engines.rime
laminar
librime
rime-data
rr
sonic-lineup
sonic-visualiser
surelog
uhdm
veilid
yosys-synlig

Maintainer pings

Flagging this as security related, it includes the fix for  CVE-2023-48230.
https://github.com/capnproto/capnproto/compare/v1.0.1...v<IP_ADDRESS>


MSSQL support :D

From the commit:

I did need to effectively comment out parts of test_13_inheritance_and_ct_tracker  because it was expecting a certain number of SQL queries (4) and the changes now bump this up to 14. The trade-off is there are a few more queries, but the results are cached, and the code is a little cleaner. Thoughts on how to resolve the tests are welcome.

Ok, so the tests show it fails for Django <1.8.
My position is that Django 1.7 is now end of life, so support isn't a huge priority, but just that's my opinion, willing to discuss how to improve this.
Also, Travis-CI doesn't have support for MSSQL yet which makes CI testing for that hard.


Allow PHP 8

Allow PHP 8 in composer.json and update CI to test with PHP 7.4 and 8.0

Many thanks for this!


How to use language model?

i am developing a realtime pvp with speech recognition.
I tried this plugin. i like it,
but now can use keyword and grammer only, how about lm?
And will you implement the training process into ue4?

language models are for free form text detection.
I didn't see much need for this, and as such omitted it.
Free form text detection has pretty poor accuracy in Pocketsphinx, in my experimentation.
In the long-term future, I would like to add in model adaption into the plugin. However, I have many more pressing things to work on (nothing to do with this plugin).
Feel free to clone the repo, and make changes to your own copy however you wish.
Then let me know, and if worthwhile, I can merge them into the main branch.

@shanecolb  This is something I would be interested in if you were will to send me the changes. I am working on a chat bot application. Thank you.

Hi Everyone,
Sorry about the delay, I have been rather busy.
I have added language model support to a demo project.
Please download the following demo project.
https://drive.google.com/open?id=0BxR5qe2wdwSLWmdBNlpBTTN5YkU
Have a look at LanguageModel_bp blueprint.
Note:  Loading language models takes a few seconds, so it is slower to initialise than the other methods.
When the Output log says 'Listening', that is when it is truly active.
I tested with the first 3 lines of X-files E01...because I felt like it ;)
"They put the time of death between eight and twelve hours ago."
"No visible cause, no sign of battery or sexuaI assault."**
"All we have is this."
Here's the result. Pretty decent, and the faults were close phonetic matches.
I think i`ll definitely look at adding this in to the main code base for the next release.
LogBlueprintUserMessages: [LanguageModel_bp_C_0] they
LogBlueprintUserMessages: [LanguageModel_bp_C_0] put
LogBlueprintUserMessages: [LanguageModel_bp_C_0] the
LogBlueprintUserMessages: [LanguageModel_bp_C_0] time
LogBlueprintUserMessages: [LanguageModel_bp_C_0] of
LogBlueprintUserMessages: [LanguageModel_bp_C_0] death
LogBlueprintUserMessages: [LanguageModel_bp_C_0] between
LogBlueprintUserMessages: [LanguageModel_bp_C_0] a
LogBlueprintUserMessages: [LanguageModel_bp_C_0] twelve
LogBlueprintUserMessages: [LanguageModel_bp_C_0] hours
LogBlueprintUserMessages: [LanguageModel_bp_C_0] ago
LogBlueprintUserMessages: [LanguageModel_bp_C_0] the
LogBlueprintUserMessages: [LanguageModel_bp_C_0] visible
LogBlueprintUserMessages: [LanguageModel_bp_C_0] cause
LogBlueprintUserMessages: [LanguageModel_bp_C_0] those
LogBlueprintUserMessages: [LanguageModel_bp_C_0] honest
LogBlueprintUserMessages: [LanguageModel_bp_C_0] battery
LogBlueprintUserMessages: [LanguageModel_bp_C_0] or
LogBlueprintUserMessages: [LanguageModel_bp_C_0] sexual
LogBlueprintUserMessages: [LanguageModel_bp_C_0] assault
LogBlueprintUserMessages: [LanguageModel_bp_C_0] all
LogBlueprintUserMessages: [LanguageModel_bp_C_0] we
LogBlueprintUserMessages: [LanguageModel_bp_C_0] have
LogBlueprintUserMessages: [LanguageModel_bp_C_0] is
LogBlueprintUserMessages: [LanguageModel_bp_C_0] this
Let me know if there's any serious problems with getting it to work.
Cheers.
Shane

Awesome thank you man! Good work.

Hello Shane, first, thank you very much for your work ! This is awesome and seems to work pretty well.
Now, I'm trying to use it for my project where I need to have a text input in VR, but virtual keyboards are not really user friendly.
So I downloaded your demo project, didn't found any langage model integration, and found this thread, downloaded the test project and it seems to work pretty well !
Now, my problem is, I'm french and need french recognition. so I tried copying the "fr" folder from the demo project to the test project, but it didn't work. I think it's because I don't have the "language_models" folder included in it, and was wondering, if I'm right, where I could find such content ?
Thanks in advance for your help, and again, excellent work, you rock !


jax2tf is failing with __call__() missing 1 required keyword-only argument: 'train'

Hi.
I am trying to convert a Vision Transformer model to TensorFlow and serialize it as a SavedModel following the guidelines from here: https://github.com/google/jax/blob/main/jax/experimental/jax2tf/examples/README.md.
I am able to generate all the artifacts needed to run the conversion but it seems like the tf_graph (out of which one would extract the concrete function) is not getting constructed properly.
Short code snippet:
predict_fn = lambda params, inputs: model.apply(dict(params=params), inputs)

tf_fn = jax2tf.convert(
    predict_fn,
    with_gradient=False,
    enable_xla=True)

param_vars = tf.nest.map_structure(
    lambda param: tf.Variable(param, trainable=False),
    params)
tf_graph = tf.function(lambda inputs: tf_fn(param_vars, inputs),
                        autograph=False,
                        jit_compile=True)

While running signatures[tf.saved_model.DEFAULT_SERVING_SIGNATURE_DEF_KEY] = \     tf_graph.get_concrete_function(input_signatures[0]) I get into:
/usr/local/lib/python3.7/dist-packages/flax/linen/module.py in wrapped_module_method(*args, **kwargs)
    269     _context.module_stack.append(self)
    270     try:
--> 271       y = fun(self, *args, **kwargs)
    272       if _context.capture_stack:
    273         filter_fn = _context.capture_stack[-1]

TypeError: __call__() missing 1 required keyword-only argument: 'train'

Here's the Colab Notebook for reproducing this error.

@gnecula if we update the TensorFlow version to tf-nightly (2.7.0-dev20210911) then the error goes away.
Another question I wanted to ask. Is there any workaround to enable fine-tuning of the converted models? This tells me it is NOT supported but I still wanted to know.

I am glad you found that the tf-nightly fix. We only test with tf-nightly, but for reasons having to do with some details of how certain TF ops are working. I did not realize that the handling of kwargs may have changed in TF.
It is possible to fine-tune the converted models, first-order reverse-mode AD works, and it is JAX-accurate. I will remove that comment, it is stale.

Thank you very much for the clarification @gnecula! I am planning to open-source the entire work done for converting the JAX Vision Transformer models to TensorFlow SavedModel. Would you folks be open to give that a mention from the jax2tf README? Might be helpful for the community.

I'd be happy to link in the README materials that the community would find helpful. Please point me to the material that you would like linked.

Would you folks be open to give that a mention from the jax2tf README? Might be helpful for the community.

@gnecula this is the material I was talking about: https://git.io/Jz8pj. It shows how to use the jax2tf tool correctly to convert Vision Transformer models to TensorFlow SavedModel and eventually load that in hub.KerasLayer for off-the-shelf inference and fine-tuning tasks.
Let me know if you'd want to have more details. Happy to have them over to you.


between matcher for dates

mydate.isBetween(date1, date2)

With this you mean Java's Date?

Yes, LocalDate, Instant, etc

Oh, ok. So not including the deprecated java.util.Date

No

We should have a matcher
beBetween for all the types, and a shouldBeBetween/shouldNotBeBetween for all types aswell.

Should we add a tolerance value?

don't need a tolerance on dates I don't think.
On Sat, 16 Feb 2019 at 23:23, Leonardo Colman Lopes <
<EMAIL_ADDRESS>wrote:

Should we add a tolerance value?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/kotlintest/kotlintest/issues/665#issuecomment-464398917,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAtZGo8HwwSw_rZZ4DMuhyCnI2Z7nu7oks5vOJLZgaJpZM4a_QR3
.

If we don't add tolerance, the user will have some hassle when using things like LocalDateTime.now(), as this exact now may be a little bit different from the now in some time from here.
Are we ok with that?

Well you can't really use now() for testing can you.
And if you have tolerance, how much tolerance? If it's a day your tests
will fail after a day. If it's a month they'll start failing after a month.
So it doesn't really work.
On Sat, 16 Feb 2019 at 23:30, Leonardo Colman Lopes <
<EMAIL_ADDRESS>wrote:

If we don't add tolerance, the user will have some hassle when using
things like LocalDateTime.now(), as this exact now may be a little bit
different from the now in some time from here.
Are we ok with that?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/kotlintest/kotlintest/issues/665#issuecomment-464399525,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAtZGkv8A0eyNXXx9FBt54_VWiL_em4uks5vOJSlgaJpZM4a_QR3
.

Sometimes you're testing thing likes
"And the creation date should be now", because inside that testing code you use now() and such. But I guess it doesn't make sense for between

Yes I see. Maybe then it would be useful I agree, so we could add something
like that.
date.isApprox(now())
On Sat, 16 Feb 2019 at 23:54, Leonardo Colman Lopes <
<EMAIL_ADDRESS>wrote:

Sometimes you're testing thing likes
"And the creation date should be now", because inside that testing code
you use now() and such. But I guess it doesn't make sense for between
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/kotlintest/kotlintest/issues/665#issuecomment-464400894,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAtZGoPCztHFC52RLhKJKE6JC1C6FMDTks5vOJotgaJpZM4a_QR3
.

Closed as per #694


Chip: filter chips "selected" class out of sync with bound value?

minimal repro here: http://jsfiddle.net/secvkhrf/
Not sure if I'm doing something wrong, but I'm just trying to change a state value when clicking the chip, which is also bound to the chip's |selected| property.
If you try clicking the chip in the fiddle, you'll see that on the first click, the state's value did change as expected, but the --selected class did not go away, so the chip is still look selected. Subsequent clicks on the chip will then toggle both the state value and the --selected class, so they will be forever out of sync (i.e. value says false when UI displays chip as selected, and vice versa).
Am I doing something wrong? or is mdc doing something in the background that's conflicting with my onClick handler?

Its the way that chipset and chips are built that they are not very prop friendly.
If its only 1 chip, I suggest it using without chipset, otherwise use chipsetRef.MDComponent.select(chipRef.MDComponent).
Not closing this, as we should implement a more friendly API here

This should be fixed with 2.0


Don't generate Autocomplete for hide commands

I want to hide some commands.
It's hide from help message, but it does not hide from generated Autocomplete.

Thank you for raising this!
That makes sense.
Will you be able to provide a pull request for this?

@power721, what version of picocli are you using?
Looking at the code for AutoComplete, hidden commands should be skipped...
Can you show me an example that reproduces this issue?

I use 4.3.2.
        <dependency>
            <groupId>info.picocli</groupId>
            <artifactId>picocli</artifactId>
            <version>4.3.2</version>
        </dependency>

I hide the item get command:
    @Command(name = "get", hidden = true, description = "Get item from the Lockbox")
    public void get(@Parameters(description = "The key", paramLabel = "KEY") String key) {
        System.out.println(lockbox.get(key));
    }

Example output:
linux:~ # ./lockbox item
contains  create    get       remove    set       update
linux:~ # ./lockbox item
Missing required subcommand
Usage: lockbox item [COMMAND]
Manage lockbox items
Commands:
  contains  Check item exist in the Lockbox
  create    Create item in the Lockbox
  remove    Remove item from the Lockbox
  set       Set item in the Lockbox
  update    Update item in the Lockbox

Fixed in master.
This will be included in the upcoming picocli 4.4 release.
Thank you again for raising this!


Add allow_tagged param to consider tagged images

Close #2

The iteration over deleting tags is currently a linear operation. However, I don't expect people to have 10000s of tags like they do random layer SHAs, so I think it's fine. We could probably parallelize it in the future if someone complains.