Using the Shark Runtime for M1 Pro?

Hi,
I read the article about the benchmark of the M1 Max. Does the Shark Runtime also work for the M1 Pro and allows to train pytorch models like the YOLOv5 or only for inference?
Thank you very much!

Yes it works on all M1 GPUs and works for both training and inference. We haven't tested Yolov5 but please do let us know if there are any ops missing.

@powderluv
Looking to try running it but can't seem to see any guide on how to get shark setup on M1. is there such a guide and using PyTorch on/with it.
Is there such a getting started guide? or I'm probably missing something.
Thanks

Do you have a tutorial/readme on how to install dependencies an use tensorflow/pytorch with Shark?

Do you have a tutorial/readme on how to install dependencies an use tensorflow/pytorch with Shark?

Would love some instructions, too.

Been trying this with PyTorch and I quite can't seem to figure out how the PyTorch models were/can be trained on the M1 GPU

I plan to publish new binaries and a easy to use instructions in a few days. Stay tuned.

The documentation will likely be in the dshark repo and tracked with this issue https://github.com/NodLabs/dSHARK/issues/26
The APIs should work already if you can build from source and export your PYTHONPATH.

Please check dSHARK apis. It should work out the box. https://github.com/NodLabs/dSHARK


Update Get-WinUserLanguageList.md

FIxed erroneous property name (InputMethodTip changed to InputMethodTips).

Docs Build status updates of commit 1d563fb:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docset/windows/international/Get-WinUserLanguageList.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

@doctordns Please change the target branch of the PR from live to master to continue with the review. Thank you.

I have no idea why this was in the 'wrong repo. I just followed the link from: https://docs.microsoft.com/en-us/powershell/module/international/get-winuserlanguagelist  to  https://github.com/MicrosoftDocs/windows-powershell-docs/blob/live/docset/windows/international/Get-WinUserLanguageList.md to https://github.com/MicrosoftDocs/windows-powershell-docs/edit/live/docset/windows/international/Get-WinUserLanguageList.md.  If that is the wrong repo then the public page has an incorrect pointer to LIve and not Master.
I am closing this RP and raised RP #1836


Can't log in after importing account to api.getferdi.com

Imported my franz account using https://api.getferdi.com/import
Changed server to https://api.getferdi.com
Login screen says wrong mail / password, so does https://api.getferdi.com/user/login since clearing browsing data (before it said invalid CSRF token)
A second attempt at importing my account tells me that my mail adress already exists in the database therefore I believe the account should exist. Additionally, there is no option to reset the password for a ferdi account?
my email adress contains a +franz@gmail and I suspect it might not got parsed correctly

The + in your email shouldn't be a problem. I suspect that the importer didn't save your password in the correct format and thus locking you out of the account.
We'll be working on a password reset mechanism in the future (https://github.com/getferdi/server/issues/16) but if you want to you can send a mail to<EMAIL_ADDRESS>with your account's e-mail address and we can reset the password manually on the server side.

This error has now been resolved via email:
The account has been imported without the "+franz" part


Bluelink Location Requires Pin

Does the. bluelinkServicePin or Pin Token need to be added to retrieve the location?  It worked in previous version but   v1.2.0 I believe was the last that worked.
This error originated from a custom integration.
Logger: custom_components.kia_uvo.HyundaiBlueLinkAPIUSA
Source: custom_components/kia_uvo/HyundaiBlueLinkAPIUSA.py:282
Integration: Kia Uvo / Hyundai Bluelink (documentation, issues)
First occurred: January 2, 2022, 11:59:25 AM (2 occurrences)
Last logged: 12:32:15 PM
kia_uvo - Unable to get vehicle location: {'errorSubCode': 'C400_1', 'systemName': 'IDM', 'functionName': 'validatePIN', 'errorSubMessage': 'Missing Fields: blueLinkServicePin or pinToken', 'errorMessage': 'Please check to see that all required fields are populated and try again.', 'errorCode': 400, 'serviceName': 'CheckCustomerUserPin'}

Could you revert version and let us know what version still works?

On Jan 3, 2022, at 10:56 AM, Noah Swint @.***> wrote:

Does the. bluelinkServicePin or Pin Token need to be added to retrieve the location? It worked in previous version but v1.2.0 I believe was the last that worked.
This error originated from a custom integration.
Logger: custom_components.kia_uvo.HyundaiBlueLinkAPIUSA
Source: custom_components/kia_uvo/HyundaiBlueLinkAPIUSA.py:282
Integration: Kia Uvo / Hyundai Bluelink (documentation, issues)
First occurred: January 2, 2022, 11:59:25 AM (2 occurrences)
Last logged: 12:32:15 PM
kia_uvo - Unable to get vehicle location: {'errorSubCode': 'C400_1', 'systemName': 'IDM', 'functionName': 'validatePIN', 'errorSubMessage': 'Missing Fields: blueLinkServicePin or pinToken', 'errorMessage': 'Please check to see that all required fields are populated and try again.', 'errorCode': 400, 'serviceName': 'CheckCustomerUserPin'}
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.

@nswint I wanted to follow up to see if you plan to check the previous version?

Closing since no response.  Please let us know if this is ongoing.


feat: Collect values written stats

This patch adds the "values written" stat to influxdb.

[x] Well-formatted commit messages
[x] Rebased/mergeable
[x] Tests pass

@ayang64 looks like the resolve merge conflict commit title is failing the semantic pull request check.


Generation time in Order and Asset tables in Postgre are always generated as current date

Reproduce steps:

Request for certificate with Generation time 1st - 30th Apr 2020.
After certificate is created, do publish for sale.

Result

In Asset, the generationFrom and generationTo are 16th Apr (current date)

In Order, the product has the generationFrom and generationTo as 15th Apr

Expected

Should the generationTime in Asset and Order be the same as the generationTime in the certificate?

Yes this is correct observation, Asset and Order should have the same Generation To and From as the on-chain Certificate.


Windows_Task not setting the run as user correctly

Describe the problem

The windows_task resource is not setting the user run as correctly.  Instead, it puts the user name in the author field in Task Scheduler.
Software Version

Windows 10
Chef-client version 14.12.9
Replication Case

Stacktrace

```
```

Possible Solution

@sydnguyen Could you please provide me some more details like recipe, logs as it is working fine with me on Windows 10 with chef-client 14.12.9

Closing this as the problem description is lacking and requires too much replication work.


service "servicename\r" is not running container #1 when trying to call any sail internal command

Sail Version
1.19.0
Laravel Version
8.83.27
PHP Version
7.4.33
Operating System
Windows (WSL)
OS Version
22H2 (22621.1702), WSL - Ubuntu 20.04.6 LTS
Description
service "servicename\r" is not running container #1 when changing APP_SERIVICE .env variable and service name in docker-compose.yml
.env lines:
APP_NAME="Service Name"
APP_ENV=local
APP_SERVICE="servicename"
APP_KEY=some key here
APP_DEBUG=true
LOG_CHANNEL=daily
APP_URL=http://localhost

docker-compose.yml:
version: '3'
services:
    servicename:
        build:
            context: ./docker/7.4
            dockerfile: Dockerfile
            args:
                WWWGROUP: '${WWWGROUP}'
        image: sail-7.4/app
        extra_hosts:
            - 'host.docker.internal:host-gateway'
        ports:
            - '${APP_PORT:-80}:80'
            - '${VITE_PORT:-5173}:${VITE_PORT:-5173}'
        environment:
            WWWUSER: '${WWWUSER}'
            LARAVEL_SAIL: 1
            XDEBUG_MODE: '${SAIL_XDEBUG_MODE:-off}'
            XDEBUG_CONFIG: '${SAIL_XDEBUG_CONFIG:-client_host=host.docker.internal}'
        volumes:
            - '.:/var/www/html'
        networks:
            - sail
        depends_on:
            - mysql
            - redis
    mysql:
        image: 'mysql/mysql-server:5.7'
        ports:
            - '${FORWARD_DB_PORT:-3306}:3306'
        environment:
            MYSQL_ROOT_PASSWORD: '${DB_PASSWORD}'
            MYSQL_ROOT_HOST: "%"
            MYSQL_DATABASE: '${DB_DATABASE}'
            MYSQL_USER: '${DB_USERNAME}'
            MYSQL_PASSWORD: '${DB_PASSWORD}'
            MYSQL_ALLOW_EMPTY_PASSWORD: 1
        volumes:
            - 'sail-mysql:/var/lib/mysql'
            - './vendor/laravel/sail/database/mysql/create-testing-database.sh:/docker-entrypoint-initdb.d/10-create-testing-database.sh'
        networks:
            - sail
        healthcheck:
            test: ["CMD", "mysqladmin", "ping", "-p${DB_PASSWORD}"]
            retries: 3
            timeout: 5s
    redis:
        image: 'redis:alpine'
        ports:
            - '${FORWARD_REDIS_PORT:-6379}:6379'
        volumes:
            - 'sail-redis:/data'
        networks:
            - sail
        healthcheck:
            test: ["CMD", "redis-cli", "ping"]
            retries: 3
            timeout: 5s
networks:
    sail:
        driver: bridge
volumes:
    sail-mysql:
        driver: local
    sail-redis:
        driver: local

The only way I found to fix this - move APP_SERVICE variable to the end of .env file, so there no line break.
Steps To Reproduce
Install sail 1.19.0 on laravel project 8.83.27 under WSL, change laravel.test to any other service name, add APP_SERVICE="other service name" in first line of .env, try to call any sail command like sail npm install

Hi there, since this is the very first report we ever received for this I'm going on a limp and say this is probably something on your end.
Can you first please try one of the support channels below? If you can actually identify this as a bug, feel free to open up a new issue with a link to the original one and we'll gladly help you out.

Laracasts Forums
Laravel.io Forums
StackOverflow
Discord
Larachat
IRC

Thanks!

Hi @Briareos17,
I was having the same issue. I deleted my .env file and remade it, ensuring that all url's were wrapped in quotation marks i.e. APP_URL="http://localhost:4200" and this seemed to do the job.


Partner token generation for China region fails

Documentation was updated to include the China region endpoint.
However, when I try to generate a partner token (so that I can register), it fails (see below)
AUDIENCE_CN="https://fleet-api.prd.cn.vn.cloud.tesla.cn"

# Partner authentication token request
curl --request POST \
  --header 'Content-Type: application/x-www-form-urlencoded' \
  --data-urlencode 'grant_type=client_credentials' \
  --data-urlencode "client_id=$CLIENT_ID" \
  --data-urlencode "client_secret=$CLIENT_SECRET" \
  --data-urlencode 'scope=openid vehicle_device_data vehicle_cmds vehicle_charging_cmds' \
  --data-urlencode "audience=$AUDIENCE_CN" \
  'https://auth.tesla.cn/oauth2/v3/token'

Response:
HTTP/2 400
{"error":"unauthorized_client","error_description":"We don't recognize this client_id and client_secret combination","error_uri":"https://auth.tesla.cn/error/reference/a4a4919e-81b7-4c44-8c55-f4d8d682ddf2-1702318465851"}

client_id and client_secret used in the request are valid and work for NA and EU regions.
Note that the request is being sent to https://auth.tesla.cn/oauth2/v3/token. Sending the request to https://auth.tesla.com/oauth2/v3/token results in the following error:
HTTP/2 400
{"error":"invalid_request","error_description":"Invalid resource","error_uri":"https://auth.tesla.com/error/reference/396f8939-be75-4716-80a1-10b77a1c2f25-1702318621906"}

ehh @sethterashima  does this mean we need a separate .cn developer account and/or permissions?

ehh @sethterashima does this mean we need a separate .cn developer account and/or permissions?

I certainly hope now. How do we even create separate one for China? Currently, there is no way to indicate that the region for developer account.
@itsMeDavidV Can you also reproduce this problem?

@rawmean yes, unfortunately just reproduced this :/

@rawmean Have you found an answer to this issue?
Looking at localized China docs, it does say to:

Set up a third-party account at https://developer.tesla.cn/

@sethterashima  does this mean that we'll need to setup a separate developer account for use of the China region?

@rawmean Have you found an answer to this issue?
Looking at localized China docs, it does say to:

Set up a third-party account at https://developer.tesla.cn/

@sethterashima  does this mean that we'll need to setup a separate developer account for use of the China region?
This is the response I get from https://fleet-api.prd.cn.vn.cloud.tesla.cn/api/1/partner_accounts.
["error_description": , "response": <null>, "error": client not found]
na and eu registration works fine.

No, getting a parter token still fails for CN. I don't see how we can create a developer account for China.

I don't see how we can create a developer account for China.

My hunch is that China developer accounts, like China user accounts, are segregated and we could quite literally make an identical dev account for the .cn domain, would be nice if someone from Tesla could confirm/clarify this.

I don't see how we can create a developer account for China.

My hunch is that China developer accounts, like China user accounts, are segregated from the rest of the Global accounts and we could quite literally make an identical dev account for the .cn domain, would be nice if someone from Tesla could confirm/clarify this.

That's a reasonable assumption, but how would one create a dev account associated with .cn? The Tesla developer site does not provide an option for creating a deva account dedicated to China. Also, why would they require such a thing? It only adds to the complexity (maintaining separate dev accounts for the same app).
I have also reported to the fleet support email a while ago, but no response.

@sethterashima Any update on this? This is a huge issue that has affected many users for weeks now.

I have the same problem. Any news or workarounds?

@ThomasAlxDmy @derekargueta @sethterashima any update on this? Should we setup separate developer accounts at https://developer.tesla.cn/ or is there something else we need to do to have our Client ID/Secrets recognized in China?

Requesting an update on this too - an additional note is that if you try to create an account on https://auth.tesla.cn/, you are prompted to verify a phone number, and are only given the option to choose the +86 (China) country code, but it seems as if it's effectively impossible (without doing something quasi-legal) for a non-Chinese national to get a +86 phone number

@sethterashima @derekargueta @ThomasAlxDmy can we get any answer on this??? This issue was created almost 3 months ago.

Update: Tesla got back to me and clarified that the Tesla instances for NA/EU are separated from CN so we will have to create a separate dev account and app at https://developer.tesla.cn/ to support CN users
They will be updating docs on this and I will be testing this out to confirm later today.

Any updates here? I noticed the docs now say

To make calls to vehicles in China, you must create an account on https://www.tesla.cn/ and create a separate application on https://developer.tesla.cn/. This requires a +86 phone number.

Does this mean if someone doesn't have a physical presence in mainland China, they will be unable to support users in China?
If that's the case, I'm not happy about it, but I understand, but I'd appreciate some official word that the +86 phone number requirement won't be changing.

If I'm not mistaken the law in China requires all data to be stored in China so this seems to be the appropriate way for Tesla to handle this.

I understand that, but I'm specifically wondering about the +86 phone number requirement.

Get a cheap voip number if you want to circumvent, pretty sure Tesla isn't responding to this due to the legal issues.

I spent a couple hours the other day doing research trying to do just that. There's no legitimate service out there that offers Chinese phone numbers that can get text messages, that I could find. If you know of one please share :)
The only thing I could find is people offering to do it gray-market was via Telegram or WhatsApp which is not the path I'd want to take. Additionally if it's a shared phone number, or one that someone else can access, I'm concerned that would give them access to my Chinese Tesla Developer account, via account recovery. I don't have proof of this, but I don't know how the Chinese Tesla accounts work.


Set id to feature

Is there a way to set id to feature (last param of ST_AsMVT function)?
id from table is not passed to feature id.

Or int value:

Hi! You can do it in your frontend as well (Mapbox GL JS). Call your id (int or string) on the function source and then:
javacript map.addSource('counties', {   "type": "vector",   "url" || "tiles": "your url / tiles",   "promoteId": **"your id"** });

@AdriSolid For now I do it with custom function and ST_AsMVT (https://postgis.net/docs/ST_AsMVT.html) takes last param as feature_id_name, but your advice is very helpful too. Thanks!
Will leave doc link here for the community: https://docs.mapbox.com/mapbox-gl-js/style-spec/sources/#vector-promoteId

great idea @AdriSolid but that is a very Mapbox GL specific feature. I do, think this feature is very valuable to have in API itself that way it's available agnostic of which client is being used to render the tiles.

This is partially already implemented, and will be cleaned up and fixed in #380


Replace existing visibility modifier in fix_visibility

Fixes #4636
I would have liked to do something about the // FIXME: this really should be a fix for diagnostic, rather than an assist., but that would take a while and there's no reason not to fix this immediately.

bors r+
Thanks!


Remove proxyUrl and hideFromSchedule from the Session type

They haven't been used since ClimateRed in 2020

They have been marked as deprecated instead, to be removed in the next major version

Released in 4.1.0


fix lower bound on ocaml version

This PR corrects the opam file to synchronise with the opam-repository: https://github.com/ocaml/opam-repository/pull/23760

Thank you!


feat(gateway-framework): remove unused deployment id authorization checks

This PR decouples the framework's auth module from any topology types. Additionally, other minor lint warnings have been fixed.

To be honest, as we are not aiming to do a release in the following days...
I'd rather merge this as is so we don't overcomplicate this PR. Next, I would merge the network module PR and integrate it. And after that, work on this authorization issue, as it may require to do some extra work to understand how the allowlists are populated.

I have updated the PR with the updated auth checks:

Context:

A subgraph may be associated with multiple deployments.
A deployment may be associated with multiple subgraphs.

If a customer sends a request with a Subgraph ID, if this ID is in the authorized subgraphs set (or the set is empty), this request is AUTHORIZED :white_check_mark:.
If a customer sends a request with a Subgraph ID not contained in the non-empty authorized subgraphs set, the request is UNAUTHORIZED and rejected :x:.
If a customer sends a request with a Deployment ID and ANY subgraph associated with this deployment is contained in the authorized subgraphs set (or the set is empty), the request is AUTHORIZED :white_check_mark:.
If a customer sends a request with a Deployment ID and ALL subgraphs associated with this deployment are NOT contained in the non-empty authorized subgraphs set, the request is UNAUTHORIZED and rejected :x:.

I am waiting for the subgraph studio team to give me some input to move things forward.

@ IainM22 agreed with the behavior described above. We can proceed and merge this PR.


Added readme files of mine - Ankush Mallick

What type of PR is this? (check all applicable)

Check according to your PR

[x] 🚀 Added Profile

[x] ✨ Feature Request

[x] ✅ Joined Community

[x] 🌟 Stared the repo

[x] 🐛 Grammatical Error

[x] 📝 Documentation Update

[x] 🚩 Other

Description
Added readme file of mine - Ankush Mallick
Add Link of GitHub Profile
https://github.com/ankushmallick1100

please merge this pr @imaaquibali .


verify function doesn't be called if token not provide or token parse error

In Strategy.js
    if (!token) {
        return self.fail(new Error("No auth token"));
    }

        if (jwt_err) {
            return self.fail(jwt_err);
        } else {

under above two situation, verify callback will not be called.
should it uses self.error instead??

Below is how I use passport-jwt
const JWT_STRATEGY_CONFIG = {
  secretOrKey: 'little secret',
  jwtFromRequest: ExtractJwt.fromAuthHeader(),
  tokenQueryParameterName: 'access_token',
  authScheme: 'Bearer',
  session: false,
  passReqToCallback: true
};

const _onJwtStrategyAuth = function(req, jwt_payload, done) {
  User
    .findOne({id: jwt_payload.sub}, function(err, user) {
        if(err) { return done(err); }
        if(!user) { return done(null, false); }
        return done(null, _onPassportAuth(user));
    });
};

passport.use(new JwtStrategy(_.assign({}, JWT_STRATEGY_CONFIG), _onJwtStrategyAuth));

This is proper behavior.  The verify callback is only called when a JWT is properly parsed.
If you want to handle parse errors or failed auth attempts you should use the custom callback functionality you want the custom callback feature of the passport framework. This is not the responsibility of the strategy.
Closing as invalid as this is not a bug.


Miscellaneous clean and pull in trivial unmerged PRs

This is the default environment so setting test was redundant.


product-pages: Better detect matching URL on product page JS.

The product page JS detects the page to run small bundle functions
but does not work correctly with language prefixes in the pathname,
such as /es/apps, so this properly detects that.
Fixes: #5635.

Automated message from Dropbox CLA bot
@brockwhittaker, it looks like you've already signed the Dropbox CLA. Thanks!

I feel like a helper function is worthwhile here, so that we don't have to repeat brittle regexes, and so that the function makes it a bit more obvious that we're essentially ignoring language prefixes.

Yeah, that's a good idea.

Yeah, it felt like the edge of DRY and don't create excessive functions, but it seems reasonable. I created a helper function!

@brockwhittaker DRY should definitely win in this sort of situation.  Any nontrivial logic should never appear more than once if one can help it, since inevitable one ends up fixing a bug in one place and not the others.  Excessive functions is a problem that usually only is relevant when the functions are all only called once.

Merged, thanks @brockwhittaker!


Full text search (FTS) indices

Given that we have https://github.com/lancedb/tantivy-object-store ready now, we can start to integrate tantive FTS into the rust core, and offer FTS to js/python/rust bindings.
Because we need to work on a variety of storage systems, we will likely need to vendor and adapt tantivy to meet our needs. Many of the components, such as the tokenizer and scoring can be re-used as is.

Maybe worth a look when we implement this: https://github.com/huggingface/tokenizers

Updates:

Inverted index implement
BM25 implement
Rust & Python interface

To get it work as soon as possible, we haven't integrate it into the filter expression, instead, we added a new interface to execute the full text search. Here is a Python example:
import random
import lance
import pyarrow as pa
import string
import tempfile

# generate dataset
n = 1000
ids = range(n)
docs = ["".join(random.choices(string.ascii_letters, k=5)) for _ in range(n)]

id_array = pa.array(ids, type=pa.int64())
 # the inverted index supports large string array only
doc_array = pa.array(docs, type=pa.large_string())

table = pa.table({"id": id_array, "doc": doc_array})
temp_dir = tempfile.mkdtemp()
dataset = lance.write_dataset(table, temp_dir)
dataset.create_scalar_index("doc","INVERTED")

results = dataset.scanner(["id", "doc"], limit=10, full_text_query=("doc", docs[0])).to_table()
print(results)

To get it work as soon as possible, I haven't integrated it into the filter expression, instead, just added a new interface to execute the full text search, but will remove this interface once we get the parser ready. Here is a Python example:
import random
import lance
import pyarrow as pa
import string
import tempfile

# generate dataset
n = 1000
ids = range(n)
docs = ["".join(random.choices(string.ascii_letters, k=5)) for _ in range(n)]

id_array = pa.array(ids, type=pa.int64())
 # the inverted index supports large string array only
doc_array = pa.array(docs, type=pa.large_string())

table = pa.table({"id": id_array, "doc": doc_array})
temp_dir = tempfile.mkdtemp()
dataset = lance.write_dataset(table, temp_dir)
dataset.create_scalar_index("doc","INVERTED")

results = dataset.scanner(["id", "doc"], limit=10, full_text_query=("doc", docs[0])).to_table()
print(results)


'BigCalendar' is not exported from 'react-big-calendar'

'BigCalendar' is not exported from 'react-big-calendar'.
I try to set up BigCalendar on my React application but I get the following error message:
Attempted import error: 'BigCalendar' is not exported from 'react-big-calendar'.
How is this possible? I've done some tests on react-big-calendar examples and it's all good.
I am using the version 0.28.6

Please check the README. The modular export is Calendar.
import { Calendar } from 'react-big-calendar'

Unless I'm missing something, the first paragraph of the README has a link titled "DEMO and Docs", the first section of which - Getting Started - says to import BigCalendar from 'react-big-calendar':
http://jquense.github.io/react-big-calendar/examples/index.html
Something out of sync here?

I guess it's not updated for the latest version.
Check the README. https://github.com/jquense/react-big-calendar

You may not be using our latest version, of the package or the docs


nicer urls for 3bot deployer

now that we have name.com client maybe better idea to do solutionname-username.3bot.grid.tf instead of the format we do on solution expose


recovery shell: ESP mount helper and tab completion

Add a mount_esp function that mounts /dev/sdxY at /mnt/sdxY if nothing else is mounted there. If the device is already mounted, return the mountpoint. The shell completion helper only completes unmounted ESP devices.
Unmounting the device is left as an exercise to the user.  I'm interested in a reasonable argument for handling this when exiting the recovery shell, like we do for ZFS datasets.
The mount operation is possibly going to be a problem. I'm trapping errors and dumping them to the console, but that's not super graceful.
Edit:
Thinking about this more, I should log all errors via zerror and then add something to the recovery shell prompt that evaluates if has_errors is set - if it is, dump the errors to the console and then clear the flag. This lets internal functions still Do The Right Thing if they're called by a program, but still have a mechanism for exposing their error text to a human operator if needed.

I think this belongs somewhere other than -core. That file provides (or should provide) critical functions used in the primary control flow of ZBM. It seems to me this should be a standalone bin script. If there is an argument against that, maybe adding a new -helpers or -utils` library that collects this and a few other user-exposed functions might be good.
At some point, it would be good to audit the ZBM library and relocate other functions as appropriate.
Having the recovery shell dump errors right to the console, maybe colored red, would be an awesome feature.

 That file provides (or should provide) critical functions used in the primary control flow of ZBM. It seems to me this should be a standalone bin script. If there is an argument against that, maybe adding a new -helpers or -utils library that collects this and a few other user-exposed functions might be good.
At some point, it would be good to audit the ZBM library and relocate other functions as appropriate.
Having the recovery shell dump errors right to the console, maybe colored red, would be an awesome feature.

My impetus for adding mount_esp as a function was simply because mount_zfs is also a function, so adding it groups it with the other dedicated mount helper function logically and on-disk. I don't have a stronger argument for leaving it in zfsbootmenu-core.sh. If you feel that it should be a stand-alone binary, I can move it there.
Based on the list above, zreport is realistically the only one that could be moved to be outside of zfsbootmenu-core.sh. It would do well to be a stand-alone binary - the -init reference to it is only when debug logging is enabled.

If zreport is the only other function that might be split up, it's probably not worth the effort to separate things at this point. We can revisit when we ponder writing some of the tools in Java.

I've added the relevant kernel modules for mounting an ESP to the optional modules list in install-helpers.sh. There's no point in building in a helper that can't (normally) mount the filesystem.

I still don't know how these modules wind up in the release or recovery images, but including the modules explicitly if they are available seems like a good idea.

I'm not sure how they're getting in there, either. I thought we were explicitly adding them in to the release/recovery images, but that's not the case. It's probably worth it to make them a hard requirement for release/recovery images on top of the best-effort installation effort that's the default.


react-native run-ios doesn't work, but running via Xcode works

Environment
Environment:
  OS: macOS High Sierra 10.13.5
  Node: 8.11.2
  Yarn: 1.7.0
  npm: 6.2.0
  Watchman: 4.9.0
  Xcode: Xcode 9.4 Build version 9F1027a
  Android Studio: 3.1 AI-173.4819257

Packages: (wanted => installed)
  react: 16.3.1 => 16.3.1
  react-native: 0.55.4 => 0.55.4

Description
I tried running via react-native run-ios, it built successfully but the app only shows the splash screen for a brief moment before it closes itself. The packager/bundler is also stuck at :
Metro Bundler ready.

Loading dependency graph, done.

I have the same issue, could you fix it?


Is there a roadmap to support 32bit windows ?

I opened a similar issue here, which was closed: https://github.com/facebook/flow/issues/2933 .
I was told there is a main issue thread about it here: https://github.com/facebook/flow/issues/270 .
Was also told there is near to no hope :-/

@Hibou57 Thanks.


Unicode/UTF-8 Character Handling Issue in REST API "/v1/chat/completions" Endpoint

🐛 Bug
Unicode/UTF-8 Character Handling Issue in REST API "/v1/chat/completions" Endpoint
To Reproduce
Steps to reproduce the behavior:

Spawn a REST API server with a model that supports outputting CJK characters or emoji.

> curl --data '{"model":"","messages":[{"role":"user","content":"Introduce yourself with lots of emojis"}],"stream":true}' --header 'Content-Type: application/json' http://<IP_ADDRESS>:8000/v1/chat/completions

output:

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":"Hello!"},"finish_reason":"stop"}]}

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":" "},"finish_reason":"stop"}]}

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":"�"},"finish_reason":"stop"}]}

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":"�"},"finish_reason":"stop"}]}

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":"�"},"finish_reason":"stop"}]}

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":"stop"}]}

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":"�"},"finish_reason":"stop"}]}

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":"�"},"finish_reason":"stop"}]}

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":"�"},"finish_reason":"stop"}]}

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":"stop"}]}

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":" I"},"finish_reason":"stop"}]}

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":"'"},"finish_reason":"stop"}]}

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":"m"},"finish_reason":"stop"}]}

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":" just"},"finish_reason":"stop"}]}

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":" an"},"finish_reason":"stop"}]}

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":" A"},"finish_reason":"stop"}]}

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":"I"},"finish_reason":"stop"}]}

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":" assistant"},"finish_reason":"stop"}]}

...

Expected behavior
Expected output:
data: {"choices":[{"index":0,"delta":{"role":"assistant","content":"Hello!"},"finish_reason":"stop"}]}

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":" "},"finish_reason":"stop"}]}

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":"😊"},"finish_reason":"stop"}]}

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":"👋"},"finish_reason":"stop"}]}

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":" I"},"finish_reason":"stop"}]}

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":"'"},"finish_reason":"stop"}]}

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":"m"},"finish_reason":"stop"}]}

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":" just"},"finish_reason":"stop"}]}

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":" an"},"finish_reason":"stop"}]}

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":" A"},"finish_reason":"stop"}]}

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":"I"},"finish_reason":"stop"}]}

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":" assistant"},"finish_reason":"stop"}]}

...

Environment

Platform (e.g. WebGPU/Vulkan/IOS/Android/CUDA): Metal
Operating system (e.g. Ubuntu/Windows/MacOS/...): macOS
Device (e.g. iPhone 12 Pro, PC+RTX 3090, ...)
How you installed MLC-LLM (conda, source): source
How you installed TVM-Unity (pip, source): pip
Python version (e.g. 3.10): 3.11
GPU driver version (if applicable):
CUDA/cuDNN version (if applicable):
TVM Unity Hash Tag (python -c "import tvm; print('\n'.join(f'{k}: {v}' for k, v in tvm.support.libinfo().items()))", applicable if you compile models):
Any other relevant information:

Potential fix
https://github.com/mlc-ai/mlc-llm/blob/7c135b83b1c96acdfcb313a76d70ec6987f8dd87/python/mlc_chat/rest.py#L167-L181
prev_txt = ""
async for content in AsyncChatCompletionStream():
    if content:
        valid_content = content.replace('�', '')
        chunk = ChatCompletionStreamResponse(
            choices=[
                ChatCompletionResponseStreamChoice(
                    index=0,
                    delta=DeltaMessage(
                        role="assistant", content=valid_content[len(prev_txt):]
                    ),
                    finish_reason="stop",
                )
            ]
        )
        prev_txt = valid_content

Faced the same emoji issue, and your fix works. Do you want to send a PR? @wanderingmeow

Closed via #1142


On traditional Windows, don't use pv on mounted file (ddev import-db)

The Problem/Issue/Bug:
There is an obscure bug apparently in pv (used in ddev import-db inside the db container) where on Windows 11 it has a buffer overflow (link).
How this PR Solves The Problem:
On windows, don't use the mounted database export, but instead push it into the container in the way we do with --no-project-mount
Manual Testing Instructions:

[ ] Do a ddev import-db using mariadb:10.3 on a known fussy machine like tb-win11-5

Automated Testing Overview:
This has not shown up in automated testing except on the new tb-win11-5. I was able to recreate it on another Windows 11 machine by doing a full uninstall/reinstall of Docker Desktop 4.8.2
Hopefully we can get this tested on tb-win11-5 as well as other machines that weren't misbehaving.

Thankfully, this code does not need to be changed. The problem was an out-of-date WSL2 kernel, which was only a problem in Windows 11, which is why it didn't affect the older Windows test runners. The pv maintainer pointed out the Dirty Pipe Vulnerability, It did turn out that the two Windows 11 machines that demonstrated it did not have updates that would update the kernel to 5.10.102.
It turns out that the Windows WSL2 kernel is now automatically updated by Windows Update...  But only if you click the Windows Update "Advanced Options" -> "Receive updates for other Microsoft Products", which is not enabled by default.


Use meta-model's XML namespace in generation

We use the parsed XML namespace instead of the hard-wired one as the XML
namespace is, somewhat unexpectedly, prone to changes. This change also
makes it easier to copy/paste the scripts for the future versions.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

1 of 1 (100.0%)  changed or added relevant line in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 90.684%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
1431

Relevant Lines:
1578

💛  - Coveralls


fix(linter): avoid infinite loop when traverse ancestors in jest/no_conditional_expect

closes: #3325

#3330  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @mysteryven and the rest of your teammates on  Graphite


Add Send Access to imported users

Hello,
first, thanks for your work.
I would like to know if it's possible to send mail activation or mail access to clients that we just saved ?
I have tryied to add the function sendInvitation() from User plugin but it's not working i think i miss something.
could you help me ?
Thanks

Hi, yes it is possible, check this line in my plugin: https://github.com/vojtasvoboda/oc-userimportexport-plugin/blob/master/models/UserImportModel.php#L51
You can uncomment it and plugin will automatically send email. You can set this email in CMS > Backend > Settings > Users > Notification.

Hi,
sorry to bother you again, but i already tried this uncomment tip. And unfortunately, it didn't worked, Moreover, i don't have any CMS > Backend > Settings > Users > Notification menu on my user backend, and i don't know why cause my rainlab-user-plugin is up to date.
Do you have another solution to give me ?
thanks a lot for your help

Since RainLab.User 1.4.0 you have to install RainLab.Notify plugin to be able to send notifications (e-mails, SMS, ...).

Thanks a lot , you're the best !


Examples

Coverage remained the same at 100.0% when pulling ff89a61b3b28d6b0b2ec90811ba6f45288533851 on examples into 45d3b18634363b2fc0b9fd3c91eb3bea49a626dc on development.


add clear doc link

@adrien-berchet @alex4200 , how do we add the link to the doc to readme? I found it here by clicking on the little doc pass thing: https://morphology-workflows.readthedocs.io/en/latest/

Indeed it's not clear. I am going to open a PR for this.


type_id and value_offset are incorrect for sliced UnionArray

Which issue does this PR close?

Closes #2086.
Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Codecov Report

Merging #2087 (6682b4f) into master (cb7e5b0) will increase coverage by 0.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #2087      +/-   ##
==========================================
+ Coverage   83.68%   83.70%   +0.02%
==========================================
  Files         224      224
  Lines       58826    58900      +74
==========================================
+ Hits        49227    49302      +75
+ Misses       9599     9598       -1

Impacted Files
Coverage Δ

arrow/src/array/array_union.rs
91.01% <100.00%> (+0.86%)
:arrow_up:

arrow/src/datatypes/datatype.rs
64.05% <0.00%> (-0.36%)
:arrow_down:

parquet/src/arrow/record_reader/mod.rs
89.17% <0.00%> (ø)

parquet/src/arrow/arrow_reader.rs
92.76% <0.00%> (+0.09%)
:arrow_up:

arrow/src/ipc/writer.rs
85.46% <0.00%> (+0.41%)
:arrow_up:

parquet_derive/src/parquet_field.rs
66.21% <0.00%> (+0.45%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cb7e5b0...6682b4f. Read the comment docs.


Expert Picks

@nrshapiro ,
I'm taking a look at the expert picks sidebar widget. I believe we will need either

A new endpoint called /expert-picks
This endpoint would take the ID's from the admin config and load all the documents and return them

OR

The ability to load multiple documents by id - perhaps from the search endpoint
@billdami mentioned that this might already be possible using something like

?smarttext={string}   recognizes if you put in a exact document identifier
or you might be able to do like fulltext1?=documentID:(123) OR documentId:(456)

Is this currently possible? Thoughts?

@bakerac4
Bill is close.
art_id:(IJP.076.0315A || AIM.062.0193A || PSC.055.0145A || REVAPA.002.0274A)
or
art_id:(IJP.076.0315A or AIM.062.0193A or PSC.055.0145A or REVAPA.002.0274A)
|| is the same as OR, but might look nicer in your code and saves the server from having to convert or/OR to it.  (Likewise && is AND.)
in the fulltext1 parameter would be best (it works in smarttext too, but if you do include anythin unusual, smarttext will be trying to interpret it.)


[2.7] CIS scan fails on k3s v1.23 clusters.

Rancher Server Setup

Rancher version: v2.7-head(f963489)
Installation option (Docker install/Helm Chart): Docker

If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc):

Proxy/Cert Details:

Information about the Cluster

Kubernetes version: v1.23.17+k3s1
Cluster Type (Local/Downstream): Downstream
If downstream, what type of cluster? (Custom/Imported or specify provider for Hosted/Infrastructure Provider): AWS node driver

User Information

What is the role of the user logged in? : Admin

Describe the bug

On K3s v1.23 k8s cluster, cis-1.23 permissive profile scan fails with below result:

Mixed
4.1.7	Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Manual)

To Reproduce

Create a rancher server v2.7-head(f963489)
Provision node driver or custom k3s v1.23.17+k3s1 cluster(1-cp, 1-e, 1w)
Install cis-benchmark-chart 4.3.0-rc4 on the cluster.
Run scan with k3s-cis-1.23-permissive profile.

Result

Observe that 4.1.7 shows Mixed result.

Expected Result

All tests should pass.

On k3s v1.23.17+k3s1 ran v1.23 permissive profile scan and it passed successfully:


Per site nucleotide diversity

Description of feature
Implement the following command either using the nf-core module.
vcftools --vcf $vcf --site-pi --positions $locations --out $prefix

Variant file is input, it can be vcf, vcf.gz or bcf
You will need to patch in positions, it is not defined in the module. Best to combine this with bed_arg.
positions is optional, we will most of the time not have this option

This feature requested by Alex Twyford.

https://vcftools.github.io/man_latest.html
--positions <filename>
--exclude-positions <filename>

Include or exclude a set of sites on the basis of a list of positions in a file.

The position file could be used as included or excluded file. Not sure what is the best way to allow user to pass the file in.
Ideally, we should have 2 inputs positions_to_include and postions_to_exlcued.
This position file can be associated with site_pi flag, @priyanka-surana do you know what other flag needs this file?
How is this file related to bed_arg?

https://github.com/sanger-tol/variantcalling/pull/62


Board Review: Azure-DigitalTwins-Core for Java

Requesting an architectural review for a client library that targets a preview Azure service.
The Basics

Service team responsible for the client library: Azure DigitalTwins
Link to documentation describing the service:

https://review.docs.microsoft.com/en-us/azure/digital-twins/?branch=master
https://github.com/Azure/azure-digital-twins/blob/private-preview/Documentation/overview.md

API surface: https://apiview.dev/Assemblies/Review/b6ca1e24411349658b47a22abc87bbaa
Product Manager contact email: sejsha
Dev Lead contact email: drwill

About this client library

Name of the client library: azure-digitaltwins-core
Languages for this review: Java
Link to the service REST APIs: https://msazure.visualstudio.com/One/_git/Azure-IoT-DigitalTwins-Main?path=%2Fproducts%2Fswagger%2Fexternal-data-plane%2Fpreview%2F2020-05-31-preview
Link to several samples: https://github.com/Azure/azure-sdk-for-java/tree/master/sdk/digitaltwins/azure-digitaltwins-core/src/samples/java/com/azure/digitaltwins/core

Artifacts required (per language)
https://apiview.dev/Assemblies/Review/b6ca1e24411349658b47a22abc87bbaa
Champion Scenarios

See samples at https://github.com/Azure/azure-sdk-for-java/tree/master/sdk/digitaltwins/azure-digitaltwins-core/src/samples/java/com/azure/digitaltwins/core

Manage model lifecycle
Models represent some physical or logical entity, such as an IoT device, a room it is in, or a building that room is in. They are the cornerstone of digital twins. Each type of entity should be modeled using the DTDL schema. Once a schema is documented in the service, it can be known to other clients so when a digital twin is loaded, a program unfamiliar with that digital twin could interact with it in a dynamic way (e.g. UI created from a model to enable an operator to interact with the digital twin).

Note: this API is only available for bulk creation at this time. When multiple models are included, the service will validate any internal references and reject the request (think transaction) if the validation fails. Without that validation, one would need to create all the models in a specific order to avoid bad references.

Create models in bulk
List models
Decommission a model
Delete a model

Digital twins
Digital Twins are the entities in ADT that represent physical or logical devices. They can be created, updated, read, and deleted.

CRUD
Query for digital twins

Relationships
Based on the models applied to a digital twin, it can have relationships to other digital twins. Those relationships are managed independent of the digital twin (they aren't properties directly on the twin), and are 1-way (that is, a relationship created from A->B only appears on A and not on B).

Link a digital twins to another via a relationship
List relationships on a digital twin
List incoming relationships (made by other digital twins targeting this one)
Delete a relationship

Telemetry
Digital twins that aren't physical devices may need telemetry sent on their behalf. The service SDK provides an avenue for that.

Publish telemetry for a digital twin, or a component on a digital twin

Event routes
An event route lets you send event data from digital twins in Azure Digital Twins to custom-defined endpoints in your subscriptions. Three Azure services are currently supported for endpoints: Event Hub, Event Grid, and Service Bus. Each of these Azure services can be connected to other services and acts as the middleman, sending data along to final destinations such as TSI or Azure Maps for whatever processing you need.

CRUD

Agenda for the review
The Azure Digital Twins Core for .NET has already been reviewed by the board a few months back. So the board is familiar with the service and the champion scenarios.
The agenda for this review is to go through the API designs and the implementation for the Java SDK.

scheduled for 9/17

[MS internal only]Video

Digital twins Java Azure SDK review action items:

Update samples of query to do while loop rather than look at just one page

Remove APIs that take/give string in favor of the strongly typed APIs only. The clazz can be a string to give the same functionality

Use Iterable rather than List for createModels APIs. Service team doesn't guarantee any ordering in this collection.

In builder, rename .httpPipeline() to .pipeline()

Take reference from search SDK on how to allow customers to set custom JsonSerializer
https://github.com/Azure/azure-sdk-for-java/blob/master/sdk/search/azure-search-documents/src/main/java/com/azure/search/documents/indexes/SearchIndexClientBuilder.java#L189

There is an autorest flag that makes required params be taken at constructor time. Srikanta will work offline with Tim to help him get this done.

Alan Zimmer offered to follow up on an Azure Core Json Patch class which is still in progress in Java's Azure Core. Ideally we'd have this class available before GA
https://github.com/Azure/azure-sdk-for-java/issues/14491

Rename "RequestOptions" to "DigitalTwinRequestOptions" to be less generic
C# and Java

Rename "ModelData" to "DigitalTwinModelData" in swagger
Johan says it should be safe to change this in the swagger since the only available SDKs are in preview.
Need to notify scripting SDK team about this upcoming change


⚠ Bofrost.lu has degraded performance

In c030cba, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 954 ms

Resolved: Bofrost.lu performance has improved in 5211e33.


missing of query stream model

Hello, I met another confusion that, in one-stream language-goal transport module, both logits and kernel are derived by passing through key_stream_one, with the other stream model (query_stream_one) unused. I don't know if it is a typo, but referring to the two-stream transport module, where the kernel is derived by passing through query_stream, I suspect it needs fixing here.
The code is here: https://github.com/cliport/cliport/blob/152610b6913a84b5d67a356e6b80c591d80d7d7a/cliport/models/streams/one_stream_transport_lang_fusion.py#L23

@huangjy-pku, thanks for catching this. This is a typo.
Fixed in https://github.com/cliport/cliport/commit/47bf8298f6b664365ab042eb7ac0bcaf5df60254.


Scaleform -- Render3DAdditive is actually Render3D (and vice versa!)

So the native that was originally called "_DRAW_SCALEFORM_MOVIE_3D_ADDITIVE" is actually just "DRAW_SCALEFORM_MOVIE_3D", so I've updated the NativeDB to reflect this. Unfortunately now we have the "_DRAW_SCALEFORM_MOVIE_3D_NON_ADDITIVE" native that I have no idea how it works. This was originally called "_DRAW_SCALEFORM_MOVIE_3D". I've tried every possible name for this unknown version:
DRAW_SCALEFORM_MOVIE_3D_FULLSCREEN
DRAW_SCALEFORM_MOVIE_3D_FULLSCREEN_MASKED
DRAW_SCALEFORM_MOVIE_3D_MASKED
DRAW_SCALEFORM_MOVIE_3D_BLEND
DRAW_SCALEFORM_MOVIE_3D_BLENDED
DRAW_SCALEFORM_MOVIE_3D_OVERLAY
DRAW_SCALEFORM_MOVIE_3D_TRANSPARENT
(and many, many more...)
I've pretty much run out of ideas for what it could be called. I'm not sure who did the original research but it would probably be a good idea to figure out what the unknown native is!
I apologize if I've created any extra work for you guys. I may do code contributions in the future, but this is just a start for me. Good luck!

As far as I understand, this is more of a NativeDB-naming problem. Since we just copy those names and don't modify them, I would suggest this topic would fit better into the gta-forums.


Timestamps cause issues with prometheus Push Gateway

When using the default prometheus push-gateway, it rejects any metrics that have timestamps on them with the error "pushed metrics must not have timestamps"
This is the url I am trying to use if Im actually misusing the push-gateway
http://<HOSTNAME>:9091/metrics/job/github_action/instance/compile_build_job/branch/${GITHUB_REF##*/}

Hello @deefdragon and thanks for checking out our github action! I should have made it more clear in the docs, but currently it only works with zapier's aggregation gateway, but we have a plan to release a new version that can handle prometheus push gateway too - stay tuned for that in the coming weeks!


dynamic table generation

django-tables2.readthedocs.org ?

hi, here is my fw builded on top of Horizon(part of OpenStack Dashboard)
i have integrated generation of tables(AngularJS), please se my doc or source code is really simple
https://github.com/michaelkuty/horizon-contrib
doc lives here http://horizon-contrib.readthedocs.org/en/latest/
main Horizon https://github.com/openstack/horizon
here is quickstart with Horizon http://docs.openstack.org/developer/horizon/quickstart.html
if you have django content types every model has own table with CRUD actions by default (generic)
without model horizon-contrib provide way for doing same as with CT
Tables has AJAX-UPDATE, Filtering(client/server), actions, Sorting pagination and many more
by default is AngularJS table , but
In the Contrib is initial implementation of React table which can handle thousand of rows without client lags
yes, with Horizon we get more than tables, workflows, modals, tabs, ..
please feel free for contact me with any questions or issues
i have created more than ten apps on these frameworks
if you want, can i create sample build on this application, for example some view can be rewritten to horizon tables
please specify your demands where you want implement tables

and provide Bootstrap 3 with compress
every CSS and JS is used as Xstatic this has many benefits for us

I have already implemented django-tables2 and they seem sufficient. Your project seems nice but I dislike the heavy integration which is kind of overkill for the one single table I need in this project...


Two richmotionsensors showing in home app

I'm not sure if this plugin has been abandoned, but I've run into something that I don't think I've noticed before.
When I activate the richmotionsensor in the config file, in the home app displays the sensor as a separate sensor instead of an accessory within the camera settings (in Home app). Within the camera settings, I see a richmotionsensor labeled Motion ServiceHome/Living (Home/Living is my mqtt topic). Incidentally, when I don't activate the richmotionsensor in homebridge config, the Motion ServiceHome/Living motion sensor still displays as an accessory for the camera.
Also the two sensors activity seem to mirror each other.
Does anyone know how to get rid of the extra sensor?
Thank you!

This is an educated guess, so someone please correct me if I'm wrong. The "normal" motion sensor is associated with the camera itself, and is listed as an accessory of the camera (as you noted). The rich motion sensor triggers off of the normal motion sensor & adds the rich message content. So it's more of an "add-on" rather than a replacement sensor.

That definitely makes sense, except what I've noticed is that you don't even need to activate richmotionsensor in the config.json file to get rich notifications. The normal motion sensor works without it. I've excluded richmotionsensor from my config file. The part that slightly annoys me is that the normal motion sensor is given an odd name by default in the home app, "Motion ServiceHome/Living" for instance. You can easily rename it in the Home app, but still.

I forgot to mention that I've excluded BOTH motionsensor and richmotionsensor from my config file and the oddly named motion sensor still shows as an accessory for the camera. That's the motion sensor I use for getting the rich notifications.

If you exclude all motion sensor accessories from the plugin config, the motion sensor still shows up, but in its settings in the Home app, there's only a "Status" option, rather than "Status and Notifications." In this setup, the sensor can be used to trigger automations, but you can't "Enable notifications on this device."

Yes, when I view the camera's phantom motion sensor setting, it only shows status. But, in the camera's settings, there is a notification settings, and in there if I turn on "Notifications on This iPhone", I will get rich notifications when the camera's phantom motion sensor detects motion, at least I think it's using the phantom motion sensor.
So normally rich notifications would be set up in the motion sensor and not in the camera even though it also works from the camera? Very confusing.

I agree, this is a confusing issue. I guess I'd rather have too many than not enough, but still...

The camera's motion sensor doesn't seem to have a "debounce" setting on the camera itself, so the plugin motion sensor "threshold" setting gives you additional control over multiple rapid motion events. That's one advantage to using the plugin for motion notifications vs. the camera.

I'm realizing that the combination of homebridge-camera-ffmpeg and homebridge-dafang-mqtt-republish accomplishes pretty much everything that this plugin does, but without this kind of oddness and other errors (like FFmpeg crashing). This may be why this plugin hasn't seen any love lately. I'm including my config, which has a Wyzecam V2 and a Wyzecam Pan, as an example.

        {
            "platform": "Camera-ffmpeg",
            "plugin_map": {
                "plugin_name": "homebridge-camera-ffmpeg"
            },
            "name": "Camera FFmpeg",
            "cameras": [
                {
                    "name": "GateCam",
                    "manufacturer": "Wyze",
                    "model": "Wyzecam V2",
                    "folder": "/home/hoobs/cameras/gatecam",
                    "motion": true,
                    "motionTimeout": 0,
                    "videoConfig": {
                        "source": "-i rtsp://<IP_ADDRESS>:8554/unicast",
                        "stillImageSource": "-i rtsp://<IP_ADDRESS>:8554/unicast -vframes 1 -r 1",
                        "maxStreams": 2,
                        "maxWidth": 768,
                        "maxHeight": 432,
                        "maxFPS": 10,
                        "vcodec": "copy",
                        "audio": true,
                        "debug": false
                    }
                },
                {
                    "name": "StudyCam",
                    "manufacturer": "Wyze",
                    "model": "Wyzecam Pan",
                    "folder": "/home/hoobs/cameras/studycam",
                    "motion": true,
                    "motionTimeout": 0,
                    "videoConfig": {
                        "source": "-i rtsp://<IP_ADDRESS>:8554/unicast",
                        "stillImageSource": "-i rtsp://<IP_ADDRESS>:8554/unicast -vframes 1 -r 1",
                        "maxStreams": 2,
                        "maxWidth": 768,
                        "maxHeight": 432,
                        "maxFPS": 10,
                        "vcodec": "copy",
                        "audio": false,
                        "debug": false
                    }
                }
            ],
            "mqtt": "<IP_ADDRESS>",
            "portmqtt": 1883,
            "usermqtt": "mqtt-user",
            "passmqtt": "mqtt-password",
            "topic": "yonder-alerts",
            "porthttp": 8089,
            "interfaceName": "eth0"
        },
        {
            "platform": "dafangMqtt",
            "plugin_map": {
                "plugin_name": "homebridge-dafang-mqtt-republish"
            },
            "name": "Dafang MQTT",
            "server": "<IP_ADDRESS>",
            "port": 1883,
            "homebridge_topic": "yonder-alerts",
            "cameras": [
                {
                    "name": "GateCam",
                    "dafang_topic": "yonder-alerts/gatecam",
                    "cooldown": 10,
                    "accessories": {
                        "recording": true,
                        "snapshot": true,
                        "reboot": true
                    }
                },
                {
                    "name": "StudyCam",
                    "dafang_topic": "yonder-alerts/studycam",
                    "cooldown": 10,
                    "accessories": {
                        "recording": true,
                        "snapshot": true,
                        "reboot": true,
                        "motorsVertical": true,
                        "motorsHorizontal": true
                    }
                }
            ]
        }

I just now reactivated richMotionSensor and started testing it with a threshold of 60000ms. This motion sensor is only available outside of the camera. I then turned on the notification straight from this sensor.

The threshold works as expected. It notifies the second it senses motion and does not send a notification if it senses anything for the next 60000ms.
There is no rich notification! It's just a standard text notification (with no picture).
Setting the notification from the accessory motion sensor (the one with the odd name) within the camera does still send rich notifications (with picture), but there is no threshold, so it sends notifications with every motion.

Lastly, I added both normal motionSensor and richMotionSensor to the configuration. It looks the same as when I set the configuration without the normal motionSensor. I couldn't find it anywhere in the home app.
I'm curious how you were able to get the thresholds to work with your rich motion sensor and still receive rich notifications.

My understanding is that this plugin does pretty much everything via MQTT, so on the camera I enable "Motion - MQTT Message" and "Motion - MQTT Video" although I only ever seem to get a snapshot of what the camera sees when it detects motion. What I really want is a rich notification that includes a video snippet of the motion, but I haven't figured out how to make that happen yet, whether with homebridge-dafang or homebridge-camera-ffmpeg + homebridge-dafang-mqtt-republish.

I'm using homebridge-camera-ffmpeg currently for a d-link ip camera, but briefly tested homebridge-dafang-mqtt-republish to replace homebridge-dafang, but went back to homebridge-dafang for one reason as far I can remember.
I couldn't get homebridge-dafang-mqtt-republish or even dafang hacks to record video to my desktop computer directly in a decent format. homebridge-dafang-mqtt-republish only communicates dafang's mqtt and doesn't do any type recompressing of video or whatever like homebridge-dafang does. It's interesting because I looked into trying to get dafang hacks to do it, but in one of the config files, it mentioned the best it could do is save 1 fps into a video file and they were also unnecessarily large in file size. How is homebridge-dafang able to do it?
I should probably revisit homebridge-dafang-mqtt-republish and homebridge camera ffmpeg for my dafang wyze cameras because eventually I'd like to to do two way audio and that will probably never be completed in homebridge-dafang.

I haven't been able to get video recording to work properly in any configuration, but I haven't even been able to get reliable video & audio from homebridge-dafang. I have other cameras too, some non-dafang, so I have to run homebridge-camera-ffmpeg anyway, and it is definitely better at handling basic video & audio than homebridge-dafang. Plus the homebridge-dafang-mqtt-republish dev is super responsive, so I think that's where I'll throw my efforts, such as they are.

I was able to configure both plugins to stream video well, but I think it's mainly because I configured the video to be super small directly in dafangs. 1280 x 720, 15fps, and 299kbps bitrate.
One thing regarding the video recordings, there is a bit of a delay before an image displays in the recording. It's black for a few seconds, but other than that, it works.
Were you able to get two way audio to work?

Never been able to get two-way audio, but I get reliable one-way audio with homebridge-camera-ffmpeg.
I can use the MQTT switch to store video recordings on the camera, in h264 format, and they're reasonably good quality although slightly slo-mo. Have you figured a way to access video recordings from the Home app?

I could not access video recordings from Home app. In fact I'm not even sure how to get back to the snapshots that are taken from rich notifications after I've closed them.
 So when I want to look at the videos from my iphone, I go into the files app and view them. I have a problem with recordings saved to the camera's sd card. First it's on the camera, so if a burglar steals the camera, the recordings are gone too. Also, at least for me, it's a pain even getting to the files. FTP freezes up and going to the web ui is also somewhat frustrating.

I agree that ffmpeg implementation of homebridge-dafang is quite buggy. Honestly i want to remove the video streaming feature from this plugin and advice to use homebridge-camera-ffmpeg since its quite stable. I am already planning a small roadmap for this plugin to fix all the issues with some new features hopefully.

I agree that ffmpeg implementation of homebridge-dafang is quite buggy. Honestly i want to remove the video streaming feature from this plugin and advice to use homebridge-camera-ffmpeg since its quite stable. I am already planning a small roadmap for this plugin to fix all the issues with some new features hopefully.


Confused

Greetings,
I'm trying to figure out how to have my SHT40 send temp and humidity to the MyAmbiance app.  I did it with my SCD40 and the eps32.  I'm pretty low skill, and so far trying to combine the code for the SHT and  the SCD40 is getting me nowhere.  I thought this library would have a way to do it, but I'm not even getting anything on the serial monitor.
Thanks.
Diane Hallinen

This library does not directly connect to MyAmbience. Please check out the Arduino BLE gadget repository and examples if you want to build your own gadget:
https://github.com/Sensirion/arduino-ble-gadget
You can of course also try to combine those libraries but this doesn't really make sense, since if a gadget already advertises data MyAmbience will directly pick it up.

@hallinen
Thank you for getting in touch and sorry for the late reply. This library is to sniff out broadcasting gadgets and go-it-yourself gadgets, which publish data with the arduino-ble-gadget library.
As @psachs mentioned, the correct library to communicate with the MyAmbience app and building a do-it-yourself gadget can be found here https://github.com/Sensirion/arduino-ble-gadget
There are also ready to use examples using SHT40 and SCD40 including setup tutorials here:
https://github.com/Sensirion/arduino-ble-gadget/tree/master/examples/Example8_SCD4x_BLE_Gadget_with_RHT
https://github.com/Sensirion/arduino-ble-gadget/blob/master/documents/SCD4x_BLE_Gadget_Tutorial.md
to combine it with the SHT40 you can use the published SHT40 arduino driver:
https://github.com/Sensirion/info#arduino-drivers-and-examples
https://github.com/Sensirion/arduino-i2c-sht4x
Have fun building great gadgets


fix: add final mark in do_clone methods to avoid warnings

Hi @andmax! Before we can merge this, we require all commits to be signed. Can you setup commit signing and remake this commit?
Info on signing is here: https://docs.github.com/articles/about-gpg/ (Using an SSH key instead of PGP is fine and generally easier to configure).
Updating the existing commit can be done with git commit --amend -S once the signing key is configured.

Hi Allison, done!  I've gpg signed the commit amending it.
Thanks,
Andre.

Great, thanks for the contribution!


docs: fixed typos and linting issues

This PR

reworked to autogenerated docs to resolve a relative path issue
fixed invalid links
fixed various typos

How to test
Run make run-web-docs and look at the squeaky clean output.

Nice work, thanks for fixing this.


regression: fix issue #134: add a product blacklist

Coverage decreased (-0.08%) to 45.778% when pulling c6871740711ed5d5ef64778747c44274b5d883f4 on calixteman:regression_issue_134 into d38551aea2b62c1273fad8e9f212bb29a82701d8 on mozilla:master.


Add definition of enum storage type

Some APIs use enum types in their definitions. The C specification does not
define the size of the integer type used for storing and passing enum type
variables.
Provide a definition matching GCC's default behavior.
Signed-off-by: Heinrich Schuchardt<EMAIL_ADDRESS>

LGTM, @jrtc27 feel free to merge that once you are OK.


Fix clippy regression

See https://rust-lang.github.io/rust-clippy/master/index.html#needless_lifetimes
Was newly introduced
Also introducing a weekly cron to detect this type of regressions

Welcome to Codecov :tada:
Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.
Thanks for integrating Codecov - We've got you covered :open_umbrella:


Kover Reports Incorrect Line/Instruction Coverage

Describe the bug
I have some pretty basic tests for an Errors wrapper object that I created. I've copied a sample of these tests below. It's really just testing each constructor. and if it sets the arguments.
But as you can see in the screenshot of coverage, with Base and IllegalEnumArg, the Line/Instruction Coverage is 100% but with every other class, it's less than 100% and I have no idea why
Errors
None
Expected behavior
The Line/Instruction Coverage for these tests should be 100%.
Reproducer
A link to your project, if it is open source. Otherwise, please try to reproduce the problem on a small project.
If this is not feasible, give an example of the code on which the problem manifests itself.
Object/Class Under Test
package com.package.messages.utils

 object Errors {
 	open class Base(message: String, val debugId: String? = null) :
 		Exception("MessageException\n  debugId: $debugId\n  message: $message")

 	class FailedToFetchDataException(message: String, debugId: String? = null) :
 		Base("Failed to get Message Data: $message", debugId)

 	class IllegalEnumArg(enumName: String, enumValues: Int) :
 		Base(
 			"Attempted to create a $enumName with an invalid index. " +
 				"Please use an index that is between 0 and ${enumValues - 1} and try again.",
 		)

 	class InvalidClientIdException(message: String, debugId: String? = null) :
 		Base("Invalid ClientID: $message", debugId)

 	class InvalidResponseException(message: String, debugId: String? = null) :
 		Base("Invalid Response: $message", debugId)

 	class ModalFailedToLoad(message: String, debugId: String? = null) :
 		Base("Modal failed to open: $message", debugId)
 }

Test
class ErrorsTest {
	@Test
	fun testBaseException() {
		val exception = Errors.Base("test_message", "test_id")

		assertEquals("MessageException\n  debugId: test_id\n  message: test_message", exception.message)
		assertEquals("test_id", exception.debugId)
	}

	@Test
	fun testFailedToFetchDataException() {
		val exception = Errors.FailedToFetchDataException("test_message", "test_id")

		assertTrue(
			exception.message?.contains("Failed to get Message Data: test_message") ?: false,
		)
		assertEquals("test_id", exception.debugId)
	}

	@Test
	fun testIllegalEnumArgException() {
		val exception = Errors.IllegalEnumArg("enum_name", 10)

		assertTrue(
			exception.message?.contains(
				"Attempted to create a enum_name with an invalid index. " +
					"Please use an index that is between 0 and 9 and try again.",
			) ?: false,
			exception.message,
		)
		assertEquals(null, exception.debugId)
	}
}

Reports

Environment

Kover Gradle Plugin version: 0.7.4
Gradle version: 8.0
Gradle Android Plugin version: 8.0.2
Kotlin project type: Kotlin/Android
Coverage Toolset (if customized in build script): Kover

@merlinpaypal For the rest of the exception classes you never use the default value of the parameter: debugId: String? = null
Please consider writing tests like
		val exception = Errors.FailedToFetchDataException("test_message")

		assertTrue(
			exception.message?.contains("Failed to get Message Data: test_message") ?: false,
		)
		assertEquals(null, exception.debugId)

Closed as answered


fix: WindowsError keyword is undefined on non-Windows platforms

WindowsError is a subclass of OSError. It is defined and raised only
in Windows. Hence the exception statement will return a
WindowsError keyword is undefined in other platforms.
Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Can you add context on where you encountered this? will help for why this was fixed.

Removed pylint ignore
Updated Description with more context on the error.


PR merge privilege

@bkragl used to have privilege to merge Boogie PRs.  But his privilege appears to have been revoked.  Any idea what caused it to happen?  Are we following some rules about who has merge privilege?

In short, I have no idea. I certainly haven't been playing with privileges. Having said that, I vaguely remember that GitHub at some point in the past year or two changed how they deal with permissions. So maybe something got messed up that way.
Either way, I would be fine to add him back.

@bkragl :  What would you like us to do?

If you can add me to the Maintain rule, that would be great.

@keyboardDrummer :  I can't figure out how to change Bernhard's role to Maintain.  Could you please do it?

If you can add me to the Maintain rule, that would be great.

Let's do that :-)

@keyboardDrummer : I can't figure out how to change Bernhard's role to Maintain. Could you please do it?

I don't have Admin rights for boogie so I cannot change collaborator roles. @shazqadeer I thought you were an Admin, if not I suggest we ask the admins to make you one.
If you are an admin, in the settings tab of the repository, you will see the "collaborators and team" option under "Access", here:

And then you can search for a user and change the role here:

I tried to navigate to that menu but when I search for @bkragl nothing pops up.  Strange!

I don't know what could have happened. I'm not aware of any action that would have removed Bernhard. In any case, I was able to add him back (with Maintain privilege -- let me know if this ought to be something different).

Thanks, @RustanLeino .


Updating the google docs with current model and making it with public view

@ahalim-intel Is that possible to update the current doc https://docs.google.com/document/d/1Ewe9Of84GkP0b2Q2PC0y9RVZNkN2WeVEagX9m99Nrzc/edit# with latest architecture and host link public in the README.md ?

Sure! We will update it soon.

Done


skip reconcile database when all the tables have been initialized

avoid executing all SQL statements to reconcile the database when all the tables have been initialized.

/cc @clyang82 @hchenxa @morvencao

this looks good, but we can do better?

check the database condition in MGH, if it is true, then skip reconcile database, in that way, we can avoid reconcile database when operator is restarted.

Is the database reconcile expected when operator is restarted?

Yes, we expected to reconcile the database when operator is restarted.

/retest

/lgtm


[RFC] Conversation Plugin in OpenSearch

RFC - Conversation Plugin in OpenSearch
Overview
The conversation plugin is to support conversation-based applications. It is designed to provide simple conversation APIs which wrap up the interactions with ml-commons (llm support) for application developer to use, that means applications only need to use these APIs to have a conversation after a model registration with ml-commons plugin.
We will have another RFC for ml-commons ReAct support.
Components
Conversation plugin

Conversation Management - manage chat sessions. Store history messages per session and all sessions’ meta data.
Index - we use OpenSearch index as data store to provide persistent memory functionality.

ML-commons plugin

Models Management - The ml-commons plugin manages all model lifecycle, provides like predict/get APIs etc.
Tools Management - The ml-commons plugin manages all built-in tools, such as a set of OpenSearch tool, like search tool, AD tool, alerting tool, ISM tool etc. We can also let users build their tools like queries to their private knowledge bases, external metrics, etc.
Prompt Template Management - Provide some built-in prompt templates for users to generate prompts. We can also let users register their own templates to use.
Embeddings - generate vector for messages through embeddings models hosted in ml-commons.
VectorDB - provide some simple vector db functions by leveraging knn index.
Text splitter - split long document to segments fitting to the input of llm. Consider making the splitting more meaningful, like section by section to have each piece meaningful.
ReAct Agent - ml-commons also takes ReAct steps to run tools and interact with llm.

Architecture

Workflow
Chat

The chatbot frontend register their own models through ml-commons API. It contains llm meta, prompt template and parameters, tools.
After getting user message, the frontend run _chat API of conversation plugin with session id and model id. The model id is required, the conversation plugin doesn’t store any default model id for the session. If no session id in the request, the conversation plugin will create a new session for this first user message that will be store as “title” for this new session.
The conversation plugin prepares the session history and combines it into the predict request to ml-commons.
The ml-commons plugin executes a ReAct process which interact with llm, then return the ReAct result.
The conversation plugin stores the pair of use/llm messages into the session, then return llm message.
The frontend gets the llm message and return it to the user.

Session Meta data
{
     "user_id": "user-0",
    "model_id": "model-0",
    "title": "the first user message",
    "created_time": timestamp
}

Message data
{
     "user_id": "user-0",
    "session_id": "session-0",
    "question": "",
    "answer": "",
    "created_time": timestamp
}

APIs
Chat
POST /_plugins/_conversation/_chat
{
     "session_id": "session-0",
    "model_id": "model-0",
    "parameters": {
        "context": "local knowledge, examples, specific plugin description etc..."
        "question": "Hello OpenSearch",
        "tools": ["MathTool", "SearchIndexTool", "SearchPipelineTool"],
        "verbose": true
    }
}

response
{
    "session_id": "session-0",
    "answer": "llm output"
}

Get session history
GET /_plugins/_conversation/history?sessionId={string}&pageSize={integer}&currentPage={integer}

//By default sorted by created_time ascendingly
response
{
     "session_id": "2dcae340-534a-457d-a9eb-8b8ee963ed9c",
    "steps": [
        {
            "question": "",
            "answer": "",
            "created_time":<PHONE_NUMBER>529
        },
        {
            "question": "",
            "answer": "",
            "created_time":<PHONE_NUMBER>116
        },
        ...
    ]
}

Get sessions
GET /_plugins/_conversation/sessions?pageSize={integer}&currentPage={integer}

//By default sorted by created_time ascendingly
response
{
    "sessions": [
        {
            "session_id": "2dcae340-534a-457d-a9eb-8b8ee963ed9c",
            "title": "",
            "created_time":<PHONE_NUMBER>529
        },
        {
            "session_id": "502919cf-d083-4d63-8a33-6047755e5a0a",
            "title": "",
            "created_time":<PHONE_NUMBER>116
        },
        ...
    ]
}

Ml-commons APIs
Register  model
POST /_plugins/_ml/models/_register?deploy=true
{
    "name": "openAI model: chat ReAct",
    "function_name": "remote",
    "version": "1.0.0",
    "description": "test model",
    "connector": {
        "http/v1": {
            "credential": {
                "api_key": "{{api_key}}"
            },
            "parameters": {
                "model": "gpt-3.5-turbo",
                "temperature": 0.1,
                "response_filter": "$.choices[0].message.content"
            },
            "headers": {
                "Authorization": "Bearer ${credential.api_key}"
            },
            "endpoint": "https://api.openai.com/v1/chat/completions",
            "http_method": "post",
            "body_template": "{ \"model\": \"${parameters.model}\",  \"messages\": [ { \"role\": \"user\",\"content\": \"${parameters.prompt}\" } ],   \"temperature\": ${parameters.temperature}, \"stop\": ${parameters.stop}}"
        }
    },
    "tools": ["MathTool", "SearchIndexTool", "SearchPipelineTool"],
}

# response
{
    "task_id": "G7uZ_YcBwpRELbgnHdG6",
    "model_id": "HLuZ_YcBwpRELbgnHdHa",
    "status": "CREATED"
}

Predict
POST /_plugins/_ml/models/model-id/_predict
{
    "parameters": {
        "examples": ["example 1", "example 2"],
        "question": "question",
        "verbose": true
    }
}

Get tools list
GET /_plugins/_ml/tools

response
{
    "tools": ["MathTool", "SearchIndexTool", "SearchPipelineTool"]
}

Get tool
GET /_plugins/_ml/tools/MathTool

response
{
    "name": "MathTool",
    "description": "It is a tool to calculate math ......"
}

Appendix
No-Code GenAI Application Creation Support

The Application Builder UI
It is designed as a drag-and-drop application workflow builder. The visual component operations are converted to REST API calls to the different backend entities through the Application Builder Backend plugin to create the corresponding function object.
The Application Builder Backend
It is responsible for creating backend components and the whole workflow by interacting with different backend entities. After the workflow setup successfully, it returns an entry point of the application to the UI for users to interact with the application.

@jngz-es how do you see search pipelines involved here? I think there's room and it's in the diagram, but it's not clear what the integration looks like. Suggestion: think about each plugin, especially ml-commons and see if there are elements that can be extracted into a separate component that can be re-usable for other purposes. Just based on the description and not knowing any more , I could see Text Splitter being a good possibility. What do you think?

Posting on this doc as well. I noticed this RFC is quite similar to one that’s already posted on this topic (#1150). I love the excitement around these ideas! However, I'm concerned that the overlap in RFCs will cause confusion in the community and make it difficult to align our development.
We would love to find a process where we can work together. The process that I’m used to in open source communities is to start with one RFC and then iterate and add feedback rather than creating multiple RFCs. This process has some benefits - it drives alignment in the open, enables the community to share and iterate on ideas, and makes the end product easy to understand and use.
My suggestion is that we adopt this approach to work together on the RFC for conversational features in OpenSearch. We greatly appreciate the feedback you've already given to the original RFC (#1150), and we'd be happy to do the work to update this RFC and continue to iterate to incorporate any other technical suggestions you have. Let us know what you think! We are excited to find ways to work together to make OpenSearch the best platform for building conversational applications.

@macohen Hi Mark, search pipeline can be a tool similar with search tool managed by ml-commons.

@jngz-es from the call today, I think we have a path forward on how we can avoid overlap and duplicate efforts using a typical open source process. 1/RFC #1150 will describe the implementation for the OpenSearch community for conversational search (both single interaction and multi-interaction "chat"), conversational memory, and the APIs associated with these functions. The spirit of this RFC is to enable natural language (using LLMs) interactions on data stored in OpenSearch. If you have suggestions on a different architecture or approach (e.g. ideas you've proposed in this separate RFC), please add that feedback to #1150 and as a community, we can debate and decide on the best approach. Once that's been decided, we'd love the community to help build this feature set to the implementation decided on in that RFC. 2/RFC #1151 should target a different theme than existing RFCs to avoid customer and community confusion, and from the discussion so far, it seems the big difference is the intention to enable building conversational agents that do not interact with data in OpenSearch (conversely, #1150 is focused on chat interfaces for data in OpenSearch). Can we change the title of #1151 to reflect this (e.g. "Multi-agent conversational platform in OpenSearch" or something that makes it clear what is being proposed that is net new to what is already being discussed in #1150 ) and explore what these use cases are, and why OpenSearch should be a platform for customers to build these applications? This would dive deeper into the net new functionality proposed and avoid overlap with the community efforts to build conversational search in #1150. It will also help the community give feedback on the approach and where this code should live. For me, the main difference between the two RFCs are: #1150 deals with conversational search with data stored in OpenSearch (e.g. RAG) and #1151 is for conversational agents that do not interact with data in OpenSearch (more similar to LangChain's general feature set). Super critical that we follow crisp open source development practices, and I appreciate working together to make sure the community can crisply focus and develop new features together.

This RFC suggests a new general conversation plugin which will be on top of ml-commons Agent Framework (#1161). #1150 mentioned the similar thing of conversation API. And #1161 is the Agent Framework RFC. Will resolve this RFC for now. Let's continue conversation API/feature topic on #1150 and Agent Framework topic on #1161


Add CodeQL Analysis workflow

Adds a GitHub Actions workflow that runs CodeQL on every push, and on a daily schedule.
Code scanning looks for vulnerabilities, such as XSS, SQL injection, etc., in your code. If it finds any new vulnerabilities it surfaces them in the PR as check annotations, and blocks the build until they’re fixed or marked as false positives. If it finds any on the repo’s default branch it displays them in the security tab.
For now you also need to be feature flagged individually to see results in the security tab (as well as having write permission on this repo) - if you drop an email to<EMAIL_ADDRESS>I can get anyone you need added.
Finally, this is an early access program that has not been released yet, so please don't share before May 6th when we’re unveiling it at GitHub Satellite.
Cc: @greysteil
Microsoft Reviewers: Open in CodeFlow

/azp run

Hey, there's a typo in this PR - the workflow file is nested under .github/workflows/workflows when it should be in .github/workflows. @jhutchings1 - can you get that fixed up for @paulcam206?

I'm going to switch this back to draft while we confirm that the autobuilder works. Otherwise, I may need a little help figuring out the manual build steps for the (many!) languages in the repo :)

I'm going to switch this back to draft while we confirm that the autobuilder works. Otherwise, I may need a little help figuring out the manual build steps for the (many!) languages in the repo :)

Happy to help :)
All of our code lives in subdirectories under source. The individual README.mds should theoretically contain build instructions. If they don't, are seemingly out of date, or just don't seem to work, LMK. All of our VS-based builds require VS2019.

android → gradle based build. should be buildable on any platform with android & cmake tooling.

gradlew build

community/reactnative → you can skip this for now
dotnet → most likely builds on Windows only (certainly some components are Windows only).

msbuild AdaptiveCards.sln

experimental → skip this directory altogether :)
ios → obviously requires MacOS to build.

codebuild -sdk iphonesimulator -configuration Release -workspace .../source/ios/AdaptiveCards/AdaptiveCards.xcworkspace -scheme ADCIOSVisualizer -destination platform=iOS Simulator,name=iPhone 7,OS=10.3.1 build

nodejs → should build anywhere nodejs works.

npm i && npx lerna bootstrap && npx lerna run build
I think this is the only platform where the autobuilder succeeds FWIW

shared/cpp/AdaptiveCardsSharedModel → Windows only.

msbuild AdaptiveCardsSharedModel.sln

tools → skip this one
uwp → Windows only.

msbuild AdaptiveCards.sln

vscode → skip
xamarin → skip

@shalinijoshi19 FYI

@paulcam Thanks for providing the additional detail here! Given the complexity of getting a build going, I'm happy to let your team riff on this. LEt me know if you need anything from my team around the analysis specifically though!

@jhutchings1 - I created a branch and authored a codeql config that builds most of our flavors (sans iOS, which may take longer to figure out). I've got a clean run of the codeql builds, but am not seeing any alerts. I suppose it's possible that there's nothing (I guess I don't really know what checkers are enabled by default), but at least for our NodeJS project, LGTM shows a number of issues (though most are pretty minor). How can I verify that things are all wired up correctly?

@jhutchings1 - I created a branch and authored a codeql config that builds most of our flavors (sans iOS, which may take longer to figure out). I've got a clean run of the codeql builds, but am not seeing any alerts. I suppose it's possible that there's nothing (I guess I don't really know what checkers are enabled by default), but at least for our NodeJS project, LGTM shows a number of issues (though most are pretty minor). How can I verify that things are all wired up correctly?

We've tuned the default rule set quite a bit so the noisier alerts that you would've seen in LGTM are off by default. As long as the actions run is completing successfully, it's doing what it's supposed to!
Since you've got this one covered, I'm going to close my PR so there's no duplication.


FormidableError: no parser found

Support plan

Which support plan is this issue covered by? (Community, Sponsor, Enterprise): Community
Currently blocking your project/work? (yes/no): Yes
Affecting a production system? (yes/no): No

Context

Node.js version: v16.15.0
Release Line of Formidable (Legacy, Current, Next): Current
Formidable exact version: 2.0.1
Environment (node, browser, native, OS): Browser
Used with (popular names of modules): fs-extra, url, sanitizeFilename, webpack

What are you trying to achieve or the steps to reproduce?

const formidable = require("formidable");

const parseFormData = (req, { maxFieldsSize, maxFileSize }) => {
	// Occurs regardless of whether I use the commented-out syntax or the active one
	// const form = new formidable.IncomingForm();
	// form.maxFieldsSize = maxFieldsSize;
	// form.maxFileSize = maxFileSize;
	// form.multiple = true;

	const form = formidable({
		multiple: true,
		maxFieldsSize,
		maxFileSize,
	});

	return new Promise((resolve, reject) => {
		form.parse(req, (err, fields, files) => {
			return err ? reject(err) : resolve({ fields, files });
		});
	});
};

What was the result you got?
FormidableError: no parser found, with a code of 1003 and an httpCode of 415.
This then gives the Express error:
RangeError [ERR_HTTP_INVALID_STATUS_CODE]: Invalid status code: 1003

What result did you expect?
I expect to be able to parse the passed file, although I am currently unsure of how to do so. I am working on an inherited project and this seems to have worked before, and researching the error message has not lead to a solution.
Any help would be greatly appreciated!

How are you sending the form ?

@GrosSacASac these functions are the parents of the function above,
/**
 * Middleware for handling HTTP requests
 */
const parseFields = (config) => (req, res) => {
	if (["get", "head"].indexOf(req.method.toLowerCase()) !== -1) {
		return Promise.resolve(parseGet(req));
	}

	const json = parseJson(req);
	if (json) {
		return Promise.resolve(json);
	}

	return parseFormData(req, config);
};

/**
 * Creates the middleware
 */
const createMiddleware = (core) => {
	const parse = parseFields(core.config("express"));

	return (req, res, next) =>
		parse(req, res)
			.then(({ fields, files }) => {
				req.fields = fields;
				req.files = files;

				next();
			})
			.catch((error) => {
				core.logger.warn(error);
				req.fields = {};
				req.files = {};

				next(error);
			});
};

const middleware = createMiddleware(core);

That is not what I asked

Could you elaborate?
You wanted to know how the form is sent, and it is sent via parseFields in the code above. The middleware variable is used by the Express Router, which sends the form:
const router = express.Router();
router.use(middleware);

parseFields is server side, how are you sending the form client side

This is the client-side code that is used:
/**
 * Writes a file.
 *
 * @param {String|VFSFile} path The path to write
 * @param {ArrayBuffer|Blob|String} data The data
 * @param {VFSMethodOptions} [options] Options
 * @returns {Promise<Number>} File size
 */
export const writefile = (adapter, mount) =>
	(path, data, options = {}) => {
		const binary = data instanceof ArrayBuffer || data instanceof Blob
			? data
			: new Blob([data], { type: "application/octet-stream" });

		return adapter.writefile(pathToObject(path), binary, options, mount);
	};

// NOTE: This is the `adapter` function that is referenced above,
// it is in another file but it is called appropriately. It is in a JSON
// format because it is exported as part of an object
writefile: ({ path }, data, options = {}) => {
		const formData = new FormData();
		formData.append("upload", data);
		formData.append("path", path);
		formData.append("options", options);

		// This is where the server-side code above is called
		return request("writefile", formData, undefined, {
			onProgress: options.onProgress,
			xhr: Boolean(options.onProgress),
		});
	},

The error means the content type is not recognized amongst the 3 that can beparse by formidable.

So I don't know what your 'request' function does but it probably does not set the correct content type header

Inside parseFormData , what  do you get if you console.log(req.headers) ?

This issue is just started in my next.js project by self. I think another library cause this bug but I still couldn't find the conflict.

@oarsoy that was my problem, the issue is the deepmerge library, which merges a FormData object into an empty object, thus losing all the data. Transitioning to the merge-deep library solved my issue, so this repository isn't the underlying cause!
Although I do recommend having some more informative warning when this is the case @GrosSacASac, just from a user's perspective. It took me a LONG time to figure out what was going wrong, only to realize that the issue wasn't even in my code.

Noted.
@oarsoy , what do you get with console.log(req.headers) ?

@GrosSacASac Can you please check what's wrong in my case? I get the same error and I've realized that as you've said, this is due to content-type: text/plain (among request headers in devtools network section). However, even adding app.use(express.text()) to my express code doesn't resolve it, so I wonder what else I should do?
This is frontend:
  const response = await axios.patch("http://localhost:1233/items/tempid", "Hi Tom", {
    headers: { "Content-Type": "text/plain" },
  });

(Although this is a patch call, I've changed the endpoint to POST too and made a request with text/plain content-type and again got this same error, so is not an issue with patch vs post).
For some reasons, in the http://localhost:1233/items/:id endpoint, I both do uploads (using Formidable) and also process text data:
app.patch("/items/:id", (req, res, next) => {
//...
  form.parse(req, (err, fields, files) => {
    console.log({ err }); // <---
    //...
  })
});

[0] {
[0]   err: FormidableError: no parser found
[0]       at IncomingForm.writeHeaders (D:\...

I want Formidable to skip/ignore the request when there are no files/fields and request is not a form-data at all. But I don't know how to get it to do that. Maybe I should check the content-type manually? Thanks in advance.

Yeah it wasn't an issue, I could both check content-type, and also I could just wrap the text in formData too using a field.

@oarsoy that was my problem, the issue is the deepmerge library, which merges a FormData object into an empty object, thus losing all the data. Transitioning to the merge-deep library solved my issue, so this repository isn't the underlying cause!
Although I do recommend having some more informative warning when this is the case @GrosSacASac, just from a user's perspective. It took me a LONG time to figure out what was going wrong, only to realize that the issue wasn't even in my code.

how did you transition to the merge-deep library


Uncaught NotFoundError: Failed to execute 'replaceChild' on 'Node': The node to be replaced is not a child of this node.

Hi,

When I try run it, it return me those error. and my inbox-item-img div is disappear.
Thanks

Hey @ssuhat, can you please attach a jsfiddle or codepen link of your problem so that I can check it. Also please let me know which version of the library you are using.
Thanks!

Closing due to inactivity.


Need Documentation update (or just tell me what I am missing)

So I was having issues getting this working via the API. But it looks like the documentation is no longer accurate since localtunnel.me isn't used anymore and loca.lt is being used.
So every time I try to start the tunnel we just hang.
const tunnel = await localtunnel({'port': p, 'local_host': '<IP_ADDRESS>', 'subdomain': subdomain});
console.log(`${main.infoTag} Tunnel has started on ${tunnel.url}`);

The console.log never fires in this scenario but it looks like we do open the tunnel as I get the landing page. when entering my public IP the page says I'm wrong.
since the only client reaching out to it is another application, I can easily manipulate the headers, but I have no need for that yet at this part of testing.
So do we have updated docs? If so please point me there, If not, could someone tell me what I am missing here?
Thanks,

Ok, so after some playing around I discovered that the tunnel does work if you do not include subdomain: subdomain in the option list. If I only set the port, the tunnel starts up and works.