add bulk import plugin

Description

Adds the Bulk import plugins in the showcase application

Fixes:
https://issues.redhat.com/browse/RHIDP-2738
Screenshot:

Test setup:

app-config changes

dynamicPlugins:
  rootDirectory: dynamic-plugins-root
  frontend:
    janus-idp.backstage-plugin-bulk-import:
      appIcons:
        - name: bulkImportIcon
          module: BulkImportPlugin
          importName: BulkImportIcon
      dynamicRoutes:
        - path: /bulk-import/repositories
          importName: BulkImportPage
          module: BulkImportPlugin
          menuItem:
            icon: bulkImportIcon
            text: Bulk import

/approve
@ciiay PTAL and LGTM if all looks good

/retest

/assign @debsmita1

Thank you @rm3l  for testing this

when logging in as guest (with permission.enabled set to true, the Bulk Import menu shows up but clicking on it returns an error page. Is that expected? Unless I missed something, I remember that it used to be hidden for users without the right permission, no? (at least last time I tested feat(bulk-import): add permission checks in bulk import UI backstage-plugins#2034 from the plugins repo:

It functions differently in the plugins repository because there, we add the BulkImportSidebarItem, which only appears in the navigation when the user is authorized to access the plugin. However, in the showcase repository, where it's configured as a dynamic plugin, we include the bulk import icon directly in the menuItem like this:
menuItem:
     icon: bulkImportIcon
     text: Bulk import

This difference is why, even when a user isn't authorized, they can still see the icon, but they cannot access the Bulk import page since it includes access checks.

once logged in (via GitHub), it looks like there are issues with the dark theme:

I can address this in a separate ticket

Thank you @rm3l for testing this

when logging in as guest (with permission.enabled set to true, the Bulk Import menu shows up but clicking on it returns an error page. Is that expected? Unless I missed something, I remember that it used to be hidden for users without the right permission, no? (at least last time I tested feat(bulk-import): add permission checks in bulk import UI backstage-plugins#2034 from the plugins repo:

It functions differently in the plugins repository because there, we add the BulkImportSidebarItem, which only appears in the navigation when the user is authorized to access the plugin. However, in the showcase repository, where it's configured as a dynamic plugin, we include the bulk import icon directly in the menuItem like this:
menuItem:
     icon: bulkImportIcon
     text: Bulk import

This difference is why, even when a user isn't authorized, they can still see the icon, but they cannot access the Bulk import page since it includes access checks.

Okay, thanks for the explanations.

once logged in (via GitHub), it looks like there are issues with the dark theme:

I can address this in a separate ticket

Sounds good. Thanks @debsmita1 !
/lgtm

@debsmita1 Looks like there are Git conflicts. Could you please rebase it? Thanks.

/lgtm

@rm3l: #1541 failed to apply on top of branch "1.3.x":
Applying: add bulk import plugin
Using index info to reconstruct a base tree...
M	dynamic-plugins.default.yaml
M	dynamic-plugins/imports/package.json
Falling back to patching base and 3-way merge...
Auto-merging dynamic-plugins/imports/package.json
CONFLICT (content): Merge conflict in dynamic-plugins/imports/package.json
Auto-merging dynamic-plugins.default.yaml
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 add bulk import plugin
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

Looking into this.. Cherry-picking it manually..


Fix camera in FX-3D

Shift zero to the canvas center.
Rotate axis so x axis directed right, y - up and z - to camera.

https://github.com/mipt-npm/dataforge-vis/blob/dev/dataforge-vis-spatial/src/jvmMain/kotlin/hep/dataforge/vis/spatial/fx/OrbitControls.kt

Implemented in dev


"Note" default Obsidian callout not found in the plugin's callout list (Windows / Linux)

Plugin does not list the "Note" callout in the Obsidian default callouts list:

(Obsidian default Note callout)

(message from the plugin)

(plugin callout list)
👀 Observed on both Windows11 and Linux (Manjaro).
📦 Obsidian version 1.2.8.

The default Question callout is missing as well. Submitting a PR to fix this.


edit item view: random order of buttons in status tab

We've observed a random order of buttons in the status tab of the edit item view:
Order of buttons in initial request:

(different) Order of buttons in a subsequent request:

Steps to reproduce the behavior:

Open the edit item view of an arbitrary DS item
check the order of buttons
logout / login again
open the edit item view again
check the order of buttons

Currently, it is not clear to me what types of actions change the order.
Expected behavior
The order should be fixed. Permanently deletion should be the last option.

Please note , that I cannot reproduce this bug in DS CRIS 7.5.


gnome.eog: Add HEIF support

Description of changes
Fixes: https://github.com/NixOS/nixpkgs/issues/87033

Things done

[x] confirmed eog can display https://github.com/strukturag/libheif/blob/38ea1ba75c7022318596c1be5bc289f0195a2bfd/examples/example.heic?raw=true
Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/heif-image-viewer/24982/4


cnet: punt unknown ip4 packets to the kernel.

Hey Keith
The idea here is that if we've made it to the ip_proto node and the protocol isn't UDP/TCP then we punt the packet to the kernel rather than drop it.
I haven't tested this yet - but would be interested to know your initial feedback.

One thing I did not check is what does punt node require as input. At this point it is getting punted from the UDP or TCP nodes and we need to make sure the mbuf data offset is what punt expects. If I remember correctly the mbuf data offset is adjusted as it passes thru IP and UDP/TCP.
Have a look and make sure punt will work from ip_proto.

as it passes thru IP and UDP/TCP.
Have a look and make sure punt will work from ip_proto.

Cool, Thanks for the pointer @KeithWiles I will have a closer look at this

It's not as straight forward as this... we never reach this point if we have a valid IP packet that isn't UDP|TCP|GTPU...
so if for example we have an OSPF packet - it's missing the filters in the ptype node:
/* Next node for each ptype, default is '0' is "pkt_drop" */
static const uint8_t p_nxt[_PTYPE_MASK + 1] __cne_cache_aligned = {
    [_L2_L3_IPV4 | CNE_PTYPE_L4_UDP]                         = PTYPE_NEXT_IP4_INPUT,
    [_L2_L3_IPV4 | CNE_PTYPE_L4_TCP]                         = PTYPE_NEXT_IP4_INPUT,
    [_L2_L3_IPV4_EXT | CNE_PTYPE_L4_UDP]                     = PTYPE_NEXT_IP4_INPUT,
    [_L2_L3_IPV4_EXT_UNK | CNE_PTYPE_L4_UDP]                 = PTYPE_NEXT_IP4_INPUT,
    [_L2_L3_IPV4 | CNE_PTYPE_L4_UDP | CNE_PTYPE_TUNNEL_GTPU] = PTYPE_NEXT_GTPU_INPUT,
};

I'm also wondering if the packet is just an IP4 packet will it be dropped for not hitting these "filters" for the next node? or do we need to add:
[_L2_L3_IPV4 ]                         = PTYPE_NEXT_IP4_INPUT,
I'm currently testing with OSPF packets which builds on top of ip4, and I was thinking about building a generic mask maybe for
[_L2_L3_IPV4 | ANY_OTHER_PROTO] then punt that packet to  PTYPE_NEXT_IP4_INPUT
of course we lose the flexibility to drop packets we don't know earlier than PTYPE_NEXT_IP4_INPUT...

I will close this PR and create a separate issue where we can track the discussion...


Change how relative fragments are handled.

Fixes #1140 as I understand it, but somebody please do check for possible regressions.
I can successfully do the following behavior:

Click one the links from the right side content bar.
Click back in your browser.
The same should work and return to the exact link in the document you got the section.

I can't think of any behavior this change would break, as both the previous and current behavior only apply to fragment links within the same page (with the href's first character being #), and other fragment links are left alone.
One thing to check, though, is the previous behavior of combining existing rel props with noopener noreferrer on all external links. The behavior I replaced it with only does so if the link is external and doesn't already specify a rel, as I figure covering basic external links is important but if the writer is going out of their way to add a rel then they should assume full control.
If we depend on the prior behavior and can't do without it, I can revert this branch to have that functionality while keeping the fix.

Restyled had some sort of auth error that doesn't look related to the PR itself:
https://restyled.io/gh/iterative/repos/dvc.org/jobs/270328/log

👍 cool stuff @rogermparent .


$sessionStorage get undefined after refreshing the page but $localStorage not

I'm using $sessionStorage to store value in session and this is my line of code
$sessionStorage.STORE_ID = '23';
. but every time I refresh the page am getting undefined when i try to retrieve the value. am retrieving by using the following method
var id = $sessionStorage.STORE_ID;
I've tried with $localStorage. it is working fine. so what is the problem with $sessionStorage. Does it has any limitations? or Do I need to add any extra code to work with $sessionStorage. Otherwise I might modify everywhere by replace that with $localStorage.
Any help please
Regards
Anson

@ansontp22 Faced the same problem. Still haven't found a solution?


Rules & other options are not configurable in the jest test matcher

Describe the ~bug~ feature
Currently the engine code has option capabilities, but those are only accessible in rule tests & users of this library are forced to run with all rules enabled. My proposal (draft PR) is to enable these options to configure the behavior of the jest matcher.
We're super excited to use this library within our codebase, but we have large amounts of code that are not accessible. My hope is that we can support the gradual adoption of these a11y rules within our code base by configuring which rules are enabled while running .toBeAccessible. Like eslint, developers can configure rules to enable/disable coverage as they adopt in large legacy repos.
expect(button).toBeAccessible({ruleIds: ['no-empty-text']});

Additionally, I've added an optional configuration that can generically override the jest matcher violation handling. This would be another option to enable adoption. For example, repos can codemod to create .toBeAccessible tests, but only throw warnings / console logs for tests that are failing.
const overrideFunction = (violations) => {
    console.warn('A11y Violations', violations);
    return [];
}

expect(button).toBeAccessible({overrideReturnFunctionality: overrideFunction});

I've already created the PR (a little out of order 😅 ) to help demonstrate use cases and to hopefully contribute forward! Please let me know what you think, and I'm open to feedback!
Context

React Native version: [e.g. 0.60]
OS: [e.g. iOS 8.1]
Test runner: [e.g. Jest]
Accessibility Engine version: [e.g. 0.30]

Hi, @emilyseibert! I think this looks awesome. 😁 No two development teams are dealing with the same context, so this gives the users more control over how they use the lib. A few thoughts:
Global options
Passing an options object to each instance of the .toBeAccessible() matcher will probably be a hassle in larger code bases. Could we make it possible to configure the engine globally? In this case, passing options to specific instances of the matcher would just override the global configs for that test.
The returnViolations option
Currently, it doesn't really make sense to support all the engine's options directly through the matcher. For example:

The option returnViolations makes sense for the engine because when this option is true, it returns an array of violations.
It's not common (or possible??) for a matcher like .toBeAccessible() to return an array of objects. If the user passes in { returnViolations: false }, their test will still fail but it will fail without a prettified error message, which just seems weird.

BUT... I think it SHOULD make sense to support all the engine's options directly through the matcher.
The only thing you'd have to do to make this possible is remove the returnViolations option, which is true by default and isn't currently exposed to the user. The engine would always return an array of violations and this if-statement wouldn't need to exist.
The ruleIds option
I think we can keep the name of this option the same (rules) and just change its value from an array of rule objects to an array of rule ids. It'll make for a cleaner API that way.
The overrideReturnFunctionality option
I like the implementation, it's just the name that isn't very clear. 🤔 Maybe:

customViolationsHandler
filterViolations

I especially like this option because it allows users to suppress errors in third-party libraries, for example.

@aryella-lacerda Thank you for the suggestions! I've made your suggested changes and set my PR (#309 ) as "Ready to review". Can you take a look? Thanks :)


Track database objects in  dataflow graph

Description
I really like the SQL integration.
However, when database objects like tables and views are created in SQL cells, other cells that depend on these might not be executed in the correct order.
import marimo

__generated_with = "0.7.20"
app = marimo.App(width="medium")

@app.cell
def __(mo, my_table):
    _df = mo.sql(
        f"""
        SELECT * FROM my_table
        """
    )
    return

@app.cell
def __(mo, my_view):
    _df = mo.sql(
        f"""
        SELECT * FROM my_view
        """
    )
    return

@app.cell
def __(mo):
    _df = mo.sql(
        f"""
        CREATE OR REPlACE TABLE my_table (a text);
        INSERT INTO my_table (VALUES ('foo'), ('bar'));
        """
    )
    return

@app.cell
def __(mo):
    _df = mo.sql(
        f"""
        CREATE VIEW my_view AS (SELECT * FROM my_table WHERE a LIKE 'f%o')
        """
    )
    return

@app.cell(hide_code=True)
def __():
    import marimo as mo
    import pandas as pd
    return mo, pd

if __name__ == "__main__":
    app.run()

Suggested solution
Include the created tables, views, etc and their references in the dataflow graph.
Alternative
As a workaround users can track the dependencies through python objects:
import marimo

__generated_with = "0.7.20"
app = marimo.App(width="medium")

@app.cell
def __(db_my_table, mo, my_table):
    db_my_table

    _df = mo.sql(
        f"""
        SELECT * FROM my_table
        """
    )
    return

@app.cell
def __(db_my_view, mo, my_view):
    db_my_view

    _df = mo.sql(
        f"""
        SELECT * FROM my_view
        """
    )
    return

@app.cell
def __(mo):
    db_my_table = None
    _df = mo.sql(
        f"""
        CREATE OR REPlACE TABLE my_table (a text);
        INSERT INTO my_table (VALUES ('foo'), ('bar'));
        """
    )
    return db_my_table,

@app.cell
def __(db_my_table, mo):
    db_my_table
    db_my_view = None
    _df = mo.sql(
        f"""
        CREATE VIEW my_view AS (SELECT * FROM my_table WHERE a LIKE 'f%o')
        """
    )
    return db_my_view,

@app.cell(hide_code=True)
def __():
    import marimo as mo
    import pandas as pd
    return mo, pd

if __name__ == "__main__":
    app.run()

For this pure SQL cells can't be used.
Even worse, this leaves the user in charge of making sure that all database objects are tracked correctly. If one is missed the reproducibility of the notebook relies on the placement of the cells (just like in jupyter notebooks).
Additional context
No response

Extracting references from the SQL is already done, so this should work for handling workflow without the verbosity
    my_view = mo.sql(
        f"""
        CREATE VIEW my_view AS (SELECT * FROM my_table WHERE a LIKE 'f%o')
        """
    )

    my_table = mo.sql(
        f"""
        CREATE OR REPlACE TABLE my_table (a text);
        INSERT INTO my_table (VALUES ('foo'), ('bar'));
        """
    )

But my_view and my_table will be None. It would be nice if the creation statements created dataframes as well.
So if CREATE OR REPlACE TABLE my_table (a text) is a standalone statement, the variable my_table is auto set if the reference is not private- to enforce that the sql table and variable matched.

Thanks for reporting. I have it on my TODO list to look into parsing SQL definitions and references more thoroughly, perhaps using sqlglot, and integrating them into the dataflow graph.

So just a note, duckdb has a native duckdb.tokenize(), and the grammar is pretty fixed: https://github.com/duckdb/duckdb/blob/main/third_party/libpg_query/grammar/statements/create.y so could directly leverage that

So just a note, duckdb has a native duckdb.tokenize(), and the grammar is pretty fixed: https://github.com/duckdb/duckdb/blob/main/third_party/libpg_query/grammar/statements/create.y so could directly leverage that

Oh nice, even better. I'll look into that.


[Ignore] groth16 private branch

[Ignore] groth16 private branch
Its snarkVM cousin may be found here https://github.com/AleoHQ/snarkVM/pull/268
Deltas

Removes requirement from Ledger and Storage that memos must be unique
Switches node server port to 14131
Removes snarkOS-parameters module
Removes hardcoded transaction size in snarkvm-consensus/src/memory_pool.rs
Updates to use Parameters::inner_circuit_id() instead of computing the ID
Removes storage of memory pool (now truly a memory pool) (previously needed this function for testnet1)

Development
Currently trialing:
rm -rf ~/.snarkOS  && cargo run --release -- --is-miner --miner-address aleo1shhq355tyaptcej65tkrweej5wth7wqg5hcqg3hf8as5s9rtrypqs8vy3u

@howardwu do we still need to keep this branch open?

Yes, we do.


Delete unmanaged AppliedManifestWork

Signed-off-by: Jian Qiu<EMAIL_ADDRESS>
record agent id in appliedmanifestwork
delete the appliedmanifestwork if it is owned by this agent but lose the management.

/hold
still need tests

Looks good to me. The only concern is that any CRD change on managed cluster may result in CRD conflict in the multiple hub scenario if the two hubs have different version.

/lgtm

/unhold


Hi

Hi i am Onur Atakan ULUSOY,
I am the CTO of Decentra Network, I think we can communicate and be collaborator.
Please contact me:
<EMAIL_ADDRESS>

Can you check your spams folder.


Add plugin for syncpack

Thanks @PeterMK85, minimal but proper!

:rocket: This pull request is included in v5.15.0. See Release 5.15.0 for release notes.
Using Knip in a commercial project? Please consider sponsoring me.


Edit Profile submit button is easy to miss

Subject: No submit button on edit profile
Description:
Just try to edit my profile
Results:
If I had a submit button, I could have actually saved my info after filling in :)
Backend version: develop-3cdcce6b
Frontend version: develop-3cdcce6b
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:95.0) Gecko/20100101 Firefox/95.0
Page: https://app.couchers.org/profile/edit
User: evertmalfait (7939)

This user found the button, but pointed out it appears below the footer if you scroll all the way down.

Hi @lucaslcode , I have the dev environment running on gitpod.
I am unable to get to /profile/edit in dev and I am unable to login using the account I created in the prod website.

Hi @xenogenesis20! Gitpod uses a local development backend. You can log in with the username "aapeli" and password "Aapeli's password", or you can create an account, and anything which would be sent as an email in the real app will be output to the backend docker container's logs. I'm not sure how to view that on gitpod, I'll have a look.

Looks like you can use the gitpod terminal: run docker ps to list the containers, find the id of the app_backend container (like 5ff7594df4ea), then run docker logs -f 5ff, where 5ff is the first few characters of the container id. Then it will be printing the backend container logs, including any emails sent for signup/login/etc.

On mobile it works really well to have the save button floating at the bottom of the screen, but if you're on a high-resolution screen, the button at the bottom is easy to miss:

Perhaps on the desktop version it would make sense to have the "save" button next to the "account settings" button as well?

On mobile it works really well to have the save button floating at the bottom of the screen, but if you're on a high-resolution screen, the button at the bottom is easy to miss:

Perhaps on the desktop version it would make sense to have the "save" button next to the "account settings" button as well?

@lucaslcode Hey there, I was able to login and poke around the codebase. I found the button in question.
The CSS class "makeStyles-buttonContainer-81" currently has position: fixed
Changing it to position: relative moves the button to be under the form.
Here you can see some screenshots:
Web view: https://tinyurl.com/y65gbxhf
Mobile view: https://tinyurl.com/y6hujmnh
I am unfamiliar with MUI and it appears that the class is applied to many buttons therefore changing it would affect others.
Hope this helps a little. I'm interested in how it can be changed without a butterfly effect. Let me know, thanks.

@AllhandsNoFeet Heya, in regards to your comment. The "account settings" button does not move when a user scrolls the page. Ideally, the save button should be sticky at the bottom so it is always visible to the user if they are only editing one field they do not have to scroll away to find it. Perhaps Lucas can give me some insight as to how it can be done, as changing it to position: sticky  does not suffice.


Support the peers property in the v1 and v2 fabric connectors

Please share the technical limitation of Caliper that you encountered.
the new fabric connector adds the concept of peer to the network configuration. This would be a useful addition to the current connectors to be able to use those peers listed for discovery
Please detail your feature idea that could alleviate the limitation.
Enhance the v1 and v2 connectors to be able to use this information. It would have to build an in-memory connection profile for use by the sdks and always ensure discovery is used
Please share some details about your use case if possible, and how the new feature would make Caliper a better performance benchmarking framework.
A dynamic connection profile is overly complicated if you have to build one by hand especially if you want to discover from a single peer, this allows you to keep all the information in a single file for simplicity.
Please share any suggestions about the new feature's code/configuration API (using formatted YAML segments or pseudo-code).
The format would be identical to the new connector format of
peers:
-   endpoint: localhost:port
    tlsCACerts:
      pem: |
          pem-data
      path: ....
    grpcOptions:
      ssl-target-name-override: peer0.myorg.example.com
      hostnameOverride: peer0.myorg.example.com

the connector will build an appropriate internal connection profile ensuring the url is correct and the structure is correct for a dynamic connection profile of the form
name: network
version: 1.0.0
client:
  organization: MyOrg
organizations:
  MyOrg
    mspid: MyOrgMSP
    peers:
    - peer0.myorg.example.com
peers:
  peer0.myorg.example.com:
    url: grpcs://host:port
    tlsCACerts:
      pem: |
          pem data
      path: path
    grpcOptions:
      ssl-target-name-override: peer0.myorg}.example.com
      hostnameOverride: peer0.myorg.example.com

Does this affect the low-level legacy connector? It already has a peers section in the connection profile format.
Won't the items of the new peers section need a name attribute to transform them to the connection profile format? Relying on grpc override settings doesn't seem very robust.

It affects all the new implementations of the connectors which includes v1 low-level + gateway and v2 gateway but I'm hoping that it will be mainly down to a single change and not having to make any changes to the connectors themselves
As the peers are under each organisation I think we could easily create unique names based on peer${index}.${mspid}


How the reader bypass the Cloudflare's protection

I noticed that the reader can read things from pages in https://openai.com which has been highly protected by Cloudflare, if only use 'puppeteer-extra-plugin-stealth', it's not enough to bypass the Cloudflare's protection.
In the source code, there's nothing to solve the captcha automatically, and no more things about the protection bypass.
What I'd like to inquire about is whether you guys have some other or more under-the-hood changes for puppeteer that make the reader not be detected by cloudflare.
We're trying to privately deploy a similar service, but are having trouble getting a close approximation in terms of accessing page content, mainly because there's no way to get around the protection.

I am still getting stuck with Cloudflare on certain sites.  This site for example still gets detected and stops the Reader from accessing the content: https://www.podbean.com/site/search/index?v={SEARCH+QUERY+HERE}.  Before I found Jina-AI, I was able to bypass this by using seleniumbase BaseCase class.  If you are trying a custom solution to avoid detection, I suggest checking out seleniumbase`.  The creator has good documentation and many YouTube tutorials on its use (Its a wrapper of Selenium).
https://seleniumbase.io/help_docs/uc_mode/#uc-mode
Does the Reader use something similar to this?  It would be nice if it could avoid detection on sites like the example I provided that is still getting caught.

Looking into the code I see that puppeteer-extra-plugin-stealth is being used by Reader.  Is the team going to be expanding to also add an option for SeleniumBase?  It would be nice to have a uc-mode option which is what works for me right now with CloudFlare.

puppeteer-extra-plugin-stealth

puppeteer-extra-plugin-stealth could be detected by Cloudflare, References: https://github.com/berstend/puppeteer-extra/issues?q=is%3Aissue+is%3Aopen+cloudflare
That' not such convinced to me if the reader just bypass the Cloudflare's protection only by this plugin, the last commit of this plugin is at least 1 year ago.
I just developed a very similar reader with all the evasions from plugin-stealth, and that doesn't work.

That might be the simple salvage function which queries the Google web cache.
https://github.com/jina-ai/reader/blob/7c5712363ce5f75589fc7eadf4e78af9a8916ebd/backend/functions/src/services/puppeteer.ts#L467-L488
It's not guaranteed to work, though.
Alternative approaches may also include querying from the Web Archive.
However the puppeteer-extra-plugin-stealth somehow doesn't work at some level with Web Archive
Setting UA to some of the famous bots, like Slackbot, GPTBot, or even GoogleSpider, sometimes also works because the site owner accepts them, but in other cases, it triggers the site to block access directly.

That might be the simple salvage function which queries the Google web cache.
https://github.com/jina-ai/reader/blob/7c5712363ce5f75589fc7eadf4e78af9a8916ebd/backend/functions/src/services/puppeteer.ts#L467-L488
It's not guaranteed to work, though.
Alternative approaches may also include querying from the Web Archive.
However the puppeteer-extra-plugin-stealth somehow doesn't work at some level with Web Archive
Setting UA to some of the famous bots, like Slackbot, GPTBot, or even GoogleSpider, sometimes also works because the site owner accepts them, but in other cases, it triggers the site to block access directly.

That might be the simple salvage function which queries the Google web cache.
https://github.com/jina-ai/reader/blob/7c5712363ce5f75589fc7eadf4e78af9a8916ebd/backend/functions/src/services/puppeteer.ts#L467-L488
It's not guaranteed to work, though.
Alternative approaches may also include querying from the Web Archive.
However the puppeteer-extra-plugin-stealth somehow doesn't work at some level with Web Archive
Setting UA to some of the famous bots, like Slackbot, GPTBot, or even GoogleSpider, sometimes also works because the site owner accepts them, but in other cases, it triggers the site to block access directly.

I found the issue, I'm trying to deploy it to the edge compute, but it seems like too many people requesting Google so it hit the rate limit. The salvage itself works fine.
Thanks for your reply :D

what both of you @nomagick @backrunner  talking about? google cache has long been closed. and it will not work anymore. Do i miss something?

@erikdemarco Previously it was available, now closed.


Cannot connect to MySQL from Workspace container

Hi,
Maybe this is a dumb question but I'm unable to connect to the MySQL container from the Workspace one with php artisan migrate :
[PDOException]
SQLSTATE[HY000] [1045] Access denied for user<EMAIL_ADDRESS>(using password: YES)

My .env file on Laravel seems good and I don't have any problem with phpMyAdmin :
DB_CONNECTION=mysql
DB_HOST=mysql
DB_PORT=3306
DB_DATABASE=theapp
DB_USERNAME=theapp
DB_PASSWORD=theapp

What am I missing ?

NVM, was a config problem


No answer received to a simple question

Hello Team,
I run the program on RHEL 8.x, and my GPU is A100 with 20GB Memory. I follow all along the installation step based on document. Then when i run this command to launch:
python generate.py --base_model=h2oai/h2ogpt-4096-llama2-7b-chat --score_model=None --langchain_mode='UserData' --user_path=user_path --share=True
The program not stopped, but also when i asked a simple question such as 'Explain yourself', it wont generate the answer. These are the log when it happened:
Using Model h2oai/h2ogpt-4096-llama2-7b-chat
Starting get_model: h2oai/h2ogpt-4096-llama2-7b-chat
/data/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:1020: FutureWarning: The use_auth_token argument is deprecated and will be removed in v5 of Transformers.
warnings.warn(
/data/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:655: FutureWarning: The use_auth_token argument is deprecated and will be removed in v5 of Transformers.
warnings.warn(
device_map: {'': 0}
/data/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:472: FutureWarning: The use_auth_token argument is deprecated and will be removed in v5 of Transformers.
warnings.warn(
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [01:27<00:00, 43.70s/it]
/data/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/transformers/utils/hub.py:374: FutureWarning: The use_auth_token argument is deprecated and will be removed in v5 of Transformers.
warnings.warn(
Model {'base_model': 'h2oai/h2ogpt-4096-llama2-7b-chat', 'tokenizer_base_model': '', 'lora_weights': '', 'inference_server': '', 'prompt_type': 'llama2', 'prompt_dict': {'promptA': '', 'promptB': '', 'PreInstruct': '[INST] ', 'PreInput': None, 'PreResponse': '[/INST]', 'terminate_response': ['[INST]', ''], 'chat_sep': ' ', 'chat_turn_sep': ' ', 'humanstr': '[INST]', 'botstr': '[/INST]', 'generates_leading_space': False, 'system_prompt': ''}, 'visible_models': None, 'h2ogpt_key': None, 'load_8bit': False, 'load_4bit': False, 'low_bit_mode': 1, 'load_half': True, 'load_gptq': '', 'load_awq': '', 'load_exllama': False, 'use_safetensors': False, 'revision': None, 'use_gpu_id': True, 'gpu_id': 0, 'compile_model': True, 'use_cache': None, 'llamacpp_dict': {'n_gpu_layers': 100, 'use_mlock': True, 'n_batch': 1024, 'n_gqa': 0, 'model_path_llama': 'https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/resolve/main/llama-2-7b-chat.ggmlv3.q8_0.bin', 'model_name_gptj': 'ggml-gpt4all-j-v1.3-groovy.bin', 'model_name_gpt4all_llama': 'ggml-wizardLM-7B.q4_2.bin', 'model_name_exllama_if_no_config': 'TheBloke/Nous-Hermes-Llama2-GPTQ'}, 'model_path_llama': 'https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/resolve/main/llama-2-7b-chat.ggmlv3.q8_0.bin', 'model_name_gptj': 'ggml-gpt4all-j-v1.3-groovy.bin', 'model_name_gpt4all_llama': 'ggml-wizardLM-7B.q4_2.bin', 'model_name_exllama_if_no_config': 'TheBloke/Nous-Hermes-Llama2-GPTQ', 'rope_scaling': {}, 'max_seq_len': None, 'exllama_dict': {}}
Running on local URL:  http://<IP_ADDRESS>:7860
2023/10/13 16:24:50 [W] [service.go:132] login to server failed: DialTcpByHttpProxy error, StatusCode [403]
Could not create share link. Please check your internet connection or our status page: https://status.gradio.app.
Started Gradio Server and/or GUI: server_name: <IP_ADDRESS> port: None
/data/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/gradio/helpers.py:818: UserWarning: Using the update method is deprecated. Simply return a new object instead, e.g. return gr.Textbox(...) instead of return gr.update(...)   warnings.warn( /data/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/gradio/components/radio.py:134: UserWarning: Using the update method is deprecated. Simply return a new object instead, e.g. return gr.Radio(...)instead ofreturn gr.Radio.update(...).   warnings.warn( The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's attention_maskto obtain reliable results. Settingpad_token_idtoeos_token_id`:2 for open-end generation.
Can you please give me some help?

The only odd thing I see is this:
2023/10/13 16:24:50 [W] [service.go:132] login to server failed: DialTcpByHttpProxy error, StatusCode [403]

As for the 7B llama-2, it's not great with documents.  I recommend mistral 7b instead if using 7b.
But your command is fine, e.g.:
(h2ogpt) jon@pseudotensor:~/h2ogpt$ python generate.py --base_model=h2oai/h2ogpt-4096-llama2-7b-chat --score_model=None --langchain_mode='UserData' --user_path=user_path --share=True

But as soon as I add some docs, 7b llama-2 gets quickly confused.

Do you have any idea what this message is about?
2023/10/13 16:24:50 [W] [service.go:132] login to server failed: DialTcpByHttpProxy error, StatusCode [403].

Hi, I've never seen before.  Could you try setting --gradio_offline_level=2 ?  Maybe it's trying to get fonts.

Perhaps your problem is more related to the doc handling of 7b?  Did you try just not using any docs first?

Hello, thanks for the response, but i haven't upload any document yet to h2ogpt, and i don't make any db yet. Do you think something is wrong with the connection maybe?

So for additional information, i just want to start simple conversation with the AI, but the answer is keep on loading and finally it got the error message from the UI

Ya that's pretty messed up.  We run h2oGPT on A100 all the time and no issues.  So some kind of issue.  I googled that warning but nothing was obvious.

Like you said before i should use mistral 7b. If i want to do it, can i just change the command running parameters with something like this?
python generate.py --base_model=h2oai/7-mistral --score_model=None --langchain_mode='UserData' --user_path=user_path --share=True --gradio_offline_level=2

python generate.py --base_model=mistralai/Mistral-7B-Instruct-v0.1 --score_model=None --langchain_mode='UserData' --user_path=user_path --share=True --gradio_offline_level=2

Sat Oct 14 01:15:51 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.30.02              Driver Version: 530.30.02    CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3090 Ti      On | 00000000:01:00.0 Off |                  Off |
|  0%   58C    P2              125W / 480W|  18549MiB / 24564MiB |      1%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce RTX 2080         On | 00000000:03:00.0 Off |                  N/A |
| 31%   44C    P8                7W / 215W|     10MiB /  8192MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      1615      G   /usr/lib/xorg/Xorg                          156MiB |
|    0   N/A  N/A      2258      G   /usr/lib/xorg/Xorg                         1515MiB |
|    0   N/A  N/A      2391      G   /usr/bin/gnome-shell                        162MiB |
|    0   N/A  N/A      5455      G   /usr/bin/nvidia-settings                      0MiB |
|    0   N/A  N/A      5940      G   ...2605455,14348786443269456662,262144      220MiB |
|    0   N/A  N/A      6976      G   gnome-control-center                          4MiB |
|    0   N/A  N/A      8291      G   ...ures=SpareRendererForSitePerProcess       35MiB |
|    0   N/A  N/A   1498418      C   python                                    16426MiB |
|    1   N/A  N/A      1615      G   /usr/lib/xorg/Xorg                            4MiB |
|    1   N/A  N/A      2258      G   /usr/lib/xorg/Xorg                            4MiB |
+---------------------------------------------------------------------------------------+

Hello, after i run the command, i straightly got this error.
python generate.py --base_model=mistralai/Mistral-7B-Instruct-v0.1 --score_model=None --langchaimode='UserData' --user_path=user_path --share=True --gradio_offline_level=2
Using Model mistralai/mistral-7b-instruct-v0.1
Starting get_model: mistralai/Mistral-7B-Instruct-v0.1
/data/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:1020: FutureWarning: Theuse_auth_tokenargument is deprecated and will be removed in v5 of Transformers.   warnings.warn( Downloading (…)lve/main/config.json: 100%|████████████████████████████████████████████████████████| 571/571 [00:00<00:00, 1.83MB/ /data/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:655: FutureWarning: Thee_auth_tokenargument is deprecated and will be removed in v5 of Transformers.   warnings.warn( Downloading (…)okenizer_config.json: 100%|████████████████████████████████████████████████████| 1.47k/1.47k [00:00<00:00, 7.36MB/ Downloading tokenizer.model: 100%|██████████████████████████████████████████████████████████████| 493k/493k [00:00<00:00, 3.46MB/ Downloading (…)/main/tokenizer.json: 100%|████████████████████████████████████████████████████| 1.80M/1.80M [00:00<00:00, 2.53MB/ Downloading (…)cial_tokens_map.json: 100%|███████████████████████████████████████████████████████| 72.0/72.0 [00:00<00:00, 397kB/ Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Traceback (most recent call last):   File "/data/h2ogpt/generate.py", line 16, in <module>     entrypoint_main()   File "/data/h2ogpt/generate.py", line 12, in entrypoint_main     H2O_Fire(main)   File "/data/h2ogpt/src/utils.py", line 64, in H2O_Fire     fire.Fire(component=component, command=args)   File "/data/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/fire/core.py", line 141, in Fire     component_trace = _Fire(component, args, parsed_flag_args, context, name)   File "/data/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire     component, remaining_args = _CallAndUpdateTrace(   File "/data/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace     component = fn(*varargs, **kwargs)   File "/data/h2ogpt/src/gen.py", line 1258, in main     model0, tokenizer0, device = get_model(reward_type=False,   File "/data/h2ogpt/src/gen.py", line 1780, in get_model     return get_hf_model(load_8bit=load_8bit,   File "/data/h2ogpt/src/gen.py", line 1945, in get_hf_model     config, model, max_seq_len = get_config(base_model,   File "/data/h2ogpt/src/gen.py", line 1384, in get_config     model = AutoModel.from_config(   File "/data/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 441, in from_coig     return model_class._from_config(config, **kwargs)   File "/data/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/transformers/modeling_utils.py", line 1190, in _from_config     model = cls(config, **kwargs)   File "/data/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 775, in init__     self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)   File "/data/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 142, in __init__     self.weight = Parameter(torch.empty((num_embeddings, embedding_dim), **factory_kwargs),   File "/data/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/torch/utils/_device.py", line 62, in __torch_function__     return func(*args, **kwargs) RuntimeError: CUDA error: CUDA-capable device(s) is/are busy or unavailable CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile withTORCH_USE_CUDA_DSA` to enable device-side assertions.
Do you think that mistral need requirement library to install first?
I dont see anything related to mistral model in the linux section

Hello, i'm just curious about the cuda version. Is it possible if we use cuda version 11.2 to run h2ogpt? because i notice your cuda spesification have CUDA version 12.1

cuda 11.7 is normal for h2ogpt install, but various binaries use different cuda versions.  The cuda version of your toolkit won't matter if you use conda way to install the env.  I'm unsure if 11.2 will work or not.

It seems like there's a CUDA-related error, indicating an issue with GPU availability or compatibility. The error message suggests that a CUDA-capable device is either busy or unavailable.
Here are a few steps you can take to troubleshoot this issue:
Check GPU Availability:
Verify that your GPU is working correctly and that CUDA is installed.
Check CUDA Installation:
Make sure you have installed the correct version of CUDA that matches your GPU and PyTorch version.
Check PyTorch Installation:
Ensure that your PyTorch installation is compatible with your CUDA version.
GPU Memory:
Check if your GPU has enough free memory to load the Mistral model.
Use CPU :
If you don't have a compatible GPU or encounter issues, you can modify the script to use the CPU instead of the GPU.
os.environ["CUDA_VISIBLE_DEVICES"] = "-1"  # Use CPU

I've tried to install the correst version of CUDA that matches my GPU and PyTorch version. It seems that the PyTorch cannot detect my GPU because of the CUDA Version didn't match, the latest version of CUDA in my GPU is 11.2 so i've tried installed pyTorch=1.10.1 that support cuda 11.1 since no PyTorch version that support cuda 11.2 based on the https://pytorch.org/get-started/previous-versions/ . And when i tried to run this code, it return False
import torch
print(torch.cuda.is_available())

For windows, in some cases people had to do:
pip install torch==2.0.0+cu117 torchvision==0.15.1+cu117 torchaudio==2.0.1 --index-url https://download.pytorch.org/whl/cu117

separately and then GPU was seen, while just:
pip install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cu117

would not give GPU as visible, but I've not seen that.

@AlfanNasution Please try with --max_seq_len=2048 or use 4096, because by default mistral will try to run with 32k context (the max from config.json)  You are probably just running out of GPU memory.


Unable to connect to mysql database

  //connect to the data base
  $db = mysqli_connect("*****","******","******","******");

  $username = $_POST['name'];
  $userstore = $_POST['store'];
  $userlati = $_POST['lati'];
  $userlongi = $_POST['longi'];
  $storecategory = $_POST['category'];
  $userpho = $_POST['phone'];
  $useremail = $_POST['email'];
  $image = $_FILES['image']['name'];

  $sql = "INSERT INTO genral (username,store_name,latitude,longitude,category,user_phone,user_email,userimage) VALUES ('$username','$userstore','$userlati','$userlongi','$storecategory','$userpho','$useremail','$image')";
  mysqli_query($db,$sql); //store data into db
  if(move_uploaded_file($_FILES['image']['tmp_name'],$target)){
      $msg = "DATA UPLOADED SUCCESFULLY";
  }else{
      $msg = "DATA NOT UPLOADED SUCCESFULLY";
  }

}
?>
PHP code to connect

Can you suggest example to connect to mysql

How to move uploaded image to folder showing error while uploading

Hi @RishiPratap. You're doing it right from my point of view. Follow the error.


Ability to swap docker with podman

Feature Suggestion

Support podman as an alternative to docker and perhaps the default tool.
Docker Desktop requires paid licensing in large organizations while Podman is a free and still open-source alternative for companies that is almost entirely compatible with the docker and docker-compose CLI commands. Podman has recently released official support for macOS and Windows in addition to its long-time support for Linux; making podman a very viable, cross-platform alternative for companies.
Possible Implementation

Introduce a new "tool" configuration for selecting Podman or Docker and then refactor the CLI to support the configured tool. For container-based vs. native-based tooling, code that currently offers a useDocker option (e.g. techdocs-cli serve) could be refactored to useContainer and then internally use the configured tool.
Context

Context provided in "Feature Suggestion".

We currently have the ContainerRunner interface that we use for executing containers. The DockerContainerRunner is the only concrete implementation of that right now. Does it seem generic enough that we could implement a PodmanContainerRunner, and is that something you'd like to contribute?

@Rugvip It seems there are at least a couple of different implementations of docker then in the code base.
I believe I could make the change to the TechDocs CLI easily enough, but I am guessing that testing the DockerContainerRunner requires a much deeper and more thorough understanding of how Backstage is using the Docker API. Would it be appropriate to submit a PR that refactors the CLI separately from the work for DockerContainerRunner, since reviews and testing could take much longer?

The DockerContainerRunner you shared
Because this code uses the Docker API and not docker commands, it may be possible to simply start podman to host a compatible API, since Podman has supported a compatible Docker API for at least a couple of years

The hard-coded commands in the TechDocs CLI for the MkDocs server
https://github.com/backstage/backstage/blob/c9a782d60922ab637d6969b4d8ad43f47bebb737/packages/techdocs-cli/src/lib/mkdocsServer.ts#L29-L58
Because this code uses docker commands, it may be possible to alias docker='podman' to route these commands through podman, since podman is almost entirely compatible with docker.

Although there are possibly workarounds for ensuring Backstage can work with podman, it would be good to codify the option with explicit, first-class support.


Added native, JSON and XML functions

Functions are available in the onembsite\onembcode\functions namespace.
Please note that functions in onembsite\onembcode\functions\HelperFunctions are for internal use only, and users should not be able to call them from their own code.

Did you test this at all?

Did you test this at all?

I've tested the functions themselves, although I just realised I forgot to test the integration :/
I'll quickly test and I'll get back to you


Strings translated by gettextCatalog in controller are not shown translated after language switch

Hello;
I'm using bootstrap tabs in my view.
$scope.tabData = [{
heading: gettextCatalog.getString("General"),
route: 'shops.detail.general'
}, {
heading: gettextCatalog.getString('Credentials'),
route: 'shops.detail.credentials'
}];
Page opens normally with true translation. But if I switch to another language, tab headings are not translated until I make refresh
How can I fix that?

Listen to 'gettextLanguageChanged' event and update translations manually or use 'translate' directive in templates.

This issue is encoutered with any use of getString. So what's the point of providing this method if it's not updated correctly. If I need to put translation in a service and not in a template, there should be a way to update the translation automatically, not by hand!

@burakkilic
$scope.tabData = function () {
  [{
    heading: gettextCatalog.getString("General"),
    route: 'shops.detail.general'
  }, {
    heading: gettextCatalog.getString('Credentials'),
    route: 'shops.detail.credentials'
  }];
};

And in your template change tabData to tabData().

@spirau Like with my example, instead of saving the result of getString at a certain point in time, save a function that will always get the string in the current language.
e.g. Instead of this...
var message = gettextCatalog.getString('Hello world');
...
setTimeout(function () {
  alert(message);
}, 5000);

do this...
var getMessage = function () {
  gettextCatalog.getString('Hello world');
};
...
setTimeout(function () {
  alert(getMessage());
}, 5000);

It adds a bit of complexity, but it gets the job done.


Rollout update to dev container configuration to research repos

We will need to roll out the updated devcontainers configuration (#43) to research repos.
We should raise a PR rather than trying to merge automatically into their repos.
We should script this process and document it. Ideally, this would be in a manually run CI action. Assuming this CI action lives in the new repo (see #46), I don't think it will be possible to have it triggered automatically on updates to the .devcontainer config though. It might be possible to run it when we tag a new docker image, but that's expected to be rare, so probably a manual trigger is fine.
For context on previous updates here's what was done thread and the repoupdater script.

For the moment, we're going to document a manual process (https://github.com/opensafely-core/codespaces-initiative/issues/51) to get a better idea of what is required to create an automated update. Once that process has been used for a bit, we may decide to automate it, at which point we should reconsider this ticket.


2024.4 support on all demos

We need a quick update on all requirements.txt
maybe it is a good time to write a simple CI/CD python code to run the demo on a server for sanity check.
-ray

@raymondlo84 demos updated. Do you mean also AI Ref Kits?

@raymondlo84 demos updated. Do you mean also AI Ref Kits?

Ideally yes

@riacheruvu @zhuo-yoyowz please check your AI Ref Kits

Closing as already addressed.


stereo sync - missing frames Error when attempt to bring up multiple ImgManip and StereoDepth

Describe the bug
I'm trying to use multiple instance of StereoDepth node (with some ImageManip nodes attached infront). However i seem to run into performance issue.
I tried using to StereoDepth node, but one of them always get stuck in giving out disparity xout. I tried to turn on the debug output it shows this:
[18443010F139BE0F00] [1.2] [68.271] [system] [info] PRINT:LeonMss: stereo sync - missing frames!
1 (1280,800)
2 (1280,800)
d1 (1280,800)
[18443010F139BE0F00] [1.2] [68.337] [system] [info] PRINT:LeonMss: stereo sync - missing frames!
1 (1280,800)
2 (1280,800)
d1 (1280,800)
[18443010F139BE0F00] [1.2] [68.403] [system] [info] PRINT:LeonMss: stereo sync - missing frames!
1 (1280,800)
2 (1280,800)
d1 (1280,800)
[18443010F139BE0F00] [1.2] [68.469] [system] [info] PRINT:LeonMss: stereo sync - missing frames!
1 (1280,800)
2 (1280,800)
d1 (1280,800)
[18443010F139BE0F00] [1.2] [68.535] [system] [info] PRINT:LeonMss: stereo sync - missing frames!
1 (1280,800)
2 (1280,800)
d1 (1280,800)
[18443010F139BE0F00] [1.2] [68.601] [system] [info] PRINT:LeonMss: stereo sync - missing frames!
1 (1280,800)
2 (1280,800)
d1 (1280,800)

To Reproduce

I am on multi_cam_support branch with FFC type board
I manage to generate a reproducible code in python. with OV9281 type sensors connected at port A,B,C. Turning on enable_2nd_stereo should reproduce the error

Link to the code https://cdn.discordapp.com/attachments/924798503270625290/1012931344235499600/manip_stereo_test_20220827.py
Expected behaviour
There should be no stereo sync issue, and both streams of stereo disparity xlink output should work smoothly.
Addtional context
https://discord.com/channels/790680891252932659/924798503270625290/1012396405429633054

@Erol444 @szabi-luxonis Created this issue as instructed. Do let me know if this is reproducible at Luxonis side

@chengguizi

if you want to perform rectification, no need to use ImageManip, can be done inside stereo node, by providing the mesh manually using loadMeshData API. By using multiple ImageManips you don't know in which order they are executed, unless there is a script node scheduling them.
Make sure all 3 cameras are synced between them. Preferably HW sync.

stereo sync - missing frames means that received frames are not synchronized.

An important note is that if I have
enable_fsin = True
enable_2nd_manip = True
enable_2nd_stereo = False # setting this to true will make time sync warning appears

The arrival time looks very normal
d1 (1280,800) 2391
cam3 (1280,800) 2392 5859.085069
cam2 (1280,800) 2394 5859.085058
cam1 (1280,800) 2394 5859.085041
d1 (1280,800) 2392
cam3 (1280,800) 2393 5859.118393
cam2 (1280,800) 2395 5859.118379
cam1 (1280,800) 2395 5859.118358
d1 (1280,800) 2393
cam3 (1280,800) 2394 5859.151697
cam2 (1280,800) 2396 5859.151687
cam1 (1280,800) 2396 5859.151671
d1 (1280,800) 2394
cam3 (1280,800) 2395 5859.185009
cam2 (1280,800) 2397 5859.184999
cam1 (1280,800) 2397 5859.184986

Another finding, if I crop the images to 400p, then I would not have a issue with the missing frame
stereo.setInputResolution(640,400)
Does this suggest performance issue with running 2 StereoDepth nodes concurrently?

Check stereo docs:
https://docs.luxonis.com/projects/api/en/latest/components/nodes/stereo_depth/#stereo-depth-fps
Subpixel + LR-check performance for 1280x720 is 34 fps.
Stereo nodes are time-sliced.

Check stereo docs: https://docs.luxonis.com/projects/api/en/latest/components/nodes/stereo_depth/#stereo-depth-fps Subpixel + LR-check performance for 1280x720 is 34 fps. Stereo nodes are time-sliced.

Ah thanks @szabi-luxonis ! Didn't think of the fps drop of subpixel + lr, had an assumption that 720p should work with 60fps, so sharing with 2 nodes would give 30fps.
I guess then this missing frame error is an expected behaviour?
Two suggestions:

Would it be possible to make it a more direct warning, say incoming stream FPS is after than StereoDepth node's capacity?
In the case of capacity saturation, i think a better load sharing mechanism could be implemented. Currently one of the shared StereoDepth node effectively stops working. The natural degradation should be, both nodes runs at a reduced frequency.

I guess then this missing frame error is an expected behaviour?

Yes. Frames are dropped, because they are not processed in-time. You may set input queue size 1 and non-blocking for both Stereo nodes.
Thanks for the suggestions. Each node is independent from each other. A separate balancer/governor can be implemented using script node.

Hi @szabi-luxonis ,
Thanks for the recommendation, I think this is no longer a bug issue then. I shall close this thread!


[tool_crash] FileSystemException: Deletion failed, OS Error: The process cannot access the file because it is being used by another process., errno = 32

Command
flutter run

Steps to Reproduce
I have a flutter project, I have run flutter clean and then trying to run the flutter run command, and this error is coming.
Logs
FileSystemException: Deletion failed, OS Error: The process cannot access the file because it is being used by another process.
, errno = 32
#0      _Directory._deleteSync (dart:io/directory_impl.dart:201:7)
#1      FileSystemEntity.deleteSync (dart:io/file_system_entity.dart:408:7)
#2      _File._deleteSync (dart:io/file_impl.dart:295:50)
#3      FileSystemEntity.deleteSync (dart:io/file_system_entity.dart:408:7)
#4      ForwardingFileSystemEntity.deleteSync (package:file/src/forwarding/forwarding_file_system_entity.dart:70:16)
#5      ErrorHandlingFileSystem.deleteIfExists (package:flutter_tools/src/base/error_handling_io.dart:82:12)
#6      ErrorHandlingFile.copySync.<anonymous closure> (package:flutter_tools/src/base/error_handling_io.dart:344:33)
#7      _runSync (package:flutter_tools/src/base/error_handling_io.dart:573:14)
#8      ErrorHandlingFile.copySync (package:flutter_tools/src/base/error_handling_io.dart:328:5)
#9      FlutterWebSdk.updateInner (package:flutter_tools/src/flutter_cache.dart:201:16)
<asynchronous suspension>
#10     CachedArtifact.update (package:flutter_tools/src/cache.dart:677:5)
<asynchronous suspension>
#11     Cache.updateAll (package:flutter_tools/src/cache.dart:546:9)
<asynchronous suspension>
#12     FlutterCommand.verifyThenRunCommand (package:flutter_tools/src/runner/flutter_command.dart:1144:7)
<asynchronous suspension>
#13     FlutterCommand.run.<anonymous closure> (package:flutter_tools/src/runner/flutter_command.dart:1043:27)
<asynchronous suspension>
#14     AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19)
<asynchronous suspension>
#15     CommandRunner.runCommand (package:args/command_runner.dart:196:13)
<asynchronous suspension>
#16     FlutterCommandRunner.runCommand.<anonymous closure> (package:flutter_tools/src/runner/flutter_command_runner.dart:284:9)
<asynchronous suspension>

[✓] Flutter (Channel stable, 2.2.3, on Microsoft Windows [Version 10.0.19042.1052], locale en-US)
    • Flutter version 2.2.3 at D:\flutter_windows_1.17.5-stable\flutter
    • Framework revision f4abaa0735 (13 days ago), 2021-07-01 12:46:11 -0700
    • Engine revision 241c87ad80
    • Dart version 2.13.4

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.2)
    • Android SDK at C:\Users\apoor\AppData\Local\Android\Sdk
    • Platform android-30, build-tools 30.0.2
    • ANDROID_HOME = C:\Users\apoor\AppData\Local\Android\Sdk
    • Java binary at: C:\Program Files\Android\Android Studio\jre\bin\java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)
    • All Android licenses accepted.

[✓] Chrome - develop for the web
    • Chrome at C:\Program Files\Google\Chrome\Application\chrome.exe

[✓] Visual Studio - develop for Windows (Visual Studio Enterprise 2019 16.8.3)
    • Visual Studio at C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise
    • Visual Studio Enterprise 2019 version 16.8.30804.86
    • Windows 10 SDK version 10.0.18362.0

[✓] Android Studio (version 4.0)
    • Android Studio at C:\Program Files\Android\Android Studio
    • Flutter plugin version 50.0.1
    • Dart plugin version 193.7547
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)

[✓] VS Code (version 1.58.0)
    • VS Code at C:\Users\apoor\AppData\Local\Programs\Microsoft VS Code
    • Flutter extension version 3.24.0

[✓] Connected device (4 available)
    • SM A505F (mobile) • RZ8M63GZWAB • android-arm64  • Android 11 (API 30)
    • Windows (desktop) • windows     • windows-x64    • Microsoft Windows [Version 10.0.19042.1052]
    • Chrome (web)      • chrome      • web-javascript • Google Chrome 90.0.4430.93
    • Edge (web)        • edge        • web-javascript • Microsoft Edge 91.0.864.67

• No issues found!

Flutter Application Metadata
Type: app
Version: 1.0.0+1
Material: true
Android X: false
Module: false
Plugin: false
Android package: null
iOS bundle identifier: null
Creation channel: stable
Creation framework version: 02c026b03cd31dd3f867e5faeb7e104cce174c5f
Plugins
path_provider-2.0.2
path_provider_linux-2.0.0
path_provider_macos-2.0.0
path_provider_windows-2.0.1
shared_preferences-2.0.6
shared_preferences_linux-2.0.0
shared_preferences_macos-2.0.0
shared_preferences_web-2.0.0
shared_preferences_windows-2.0.0

Turned out my disk space in which Flutter SDK was present got full.
I cleared some space and its now working!
Someone should add this as an error instead of the message I got.


UPDATED FOOTER IN "READ LATER" page

Kindly assign me level 2 or 3 for this task.
I have updated the FOOTER
fixes: #4703

Kindly check it @abhi03ruchi

make it like this ...

colour theme didn't matched @neeru24

@sailaja-adapa The theme is same... You can check the HOME page itself, and others too... they too have the same theme

@abhi03ruchi Done with the changes... Have taken the screenshot from DEPLOY PREVIEW. Kindly merge with level 3 if possible. Would be grateful

this doesnt look good . left and middle section theek karo . there is space .

This is about page:

This is what I made:

May I know the difference, It's same dear

@abhi03ruchi I tried my best to reduce the space to a possible extent. after this, the placement of other things is getting changed, so kindly merge the PR, it took efforts to do this much.


Menu Config

Component Name: Menu
Version: 1.0.0
Category: Config
Type: Backend
This menu configuration is used in the Header components. It also includes another component called Link.

Used on #100


package installation errors from Atom bundle (Windows 8.1 64bit)

Thanks for reporting this. You might be able to fix it by removing the ~/.julia/lib folder.

I'll close this, but feel free to reopen if it's still a problem.


#31 Lerna

Need this PR to test what is working in builds

Unlinking from the Lerna task because we need to get this new structure in first for @lordpixel


Adding dark edge lines in light mode + enabling SSAO

LGTM. Note: It's a matter of adding the commands to not trigger a batch flush in src/wasm-lib/kcl/src/engine/mod.rs - but this can be done in another PR. It's not really a source of slowness anyway since it's just ~4 commands for the initial scene setup.

And no snapshot changes, just formatting (I don't know why that update snapshot action commits formatting changes too). I'll push an empty commit and we can merge.


Create examples of config.js reference for local testing

@klutchell Hacky but we have to make do, since for each DUT available to be tested in testbot-rig fleet. We need to provide test assets to them as well. So this is is probably the cleanest way to do what we needed to do.  https://github.com/balena-os/leviathan/blob/c3adc607b74edc395b9855b071ce4cee473b832a/documentation/config-reference.md#L156


Fix panic when building with recent rust versions.

Currently a plugin that calls register_architecture() when loaded by BN because rustc doesn't allow memory allocated by zeroed() to be uninitialized. This change fixes that by using MaybeUninit instead of zeroed().

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Can you possibly share the plugin you're working on so I can test this?  Not having an arch to test is what's preventing me from approving this.


robot lose the connection with tags when it trying to go parallel

when the robot is in front of the charging station it can detect the 3 fiducials markers and its position is parallel to the station. it starts to rotate to the left then it displays this message:
[INFO] [1653488377.911862, 903.320000] : Attempt with 1 try
[WARN] [1653488377.914467, 903.322000]: Status: [PREDOCK] | start docking
[INFO] [1653488377.919384483,<PHONE_NUMBER>00]: ObstacleObserver received autodock status: 2
[WARN] [1653488378.454973, 903.827000]: Search failure: fiducial_10, from base_footprint
[WARN] [1653488378.458897, 903.831000]: Status: [PREDOCK] | Try to set yaw of one marker
[INFO] [1653488378.463162517,<PHONE_NUMBER>00]: ObstacleObserver received autodock status: 2
[WARN] [1653488379.074444, 904.336000]: Search failure: fiducial_10, from base_footprint
[WARN] [1653488379.129678, 904.388000]: Rotate with right marker: fiducial_11
[WARN] [1653488379.132816, 904.391000]: Status: [PREDOCK] | Rotate robot: -0.02 rad
[INFO] [1653488379.137920379,<PHONE_NUMBER>00] : ObstacleObserver received autodock status : 2
Current difference of x, y, yaw: -0.00 | -0.00 | -0.02
[WARN] [1653488379.217710, 904.471000] : Finished running the robot
[INFO] [1653488379.218855, 904.472000]: Both markers are detected, execute pre-storage loop.
[WARN] [1653488379.758596, 904.973000]: Search failure: fiducial_10, from base_footprint
[ERROR] [1653488379.761544, 904.975000]: No detection of marker on two sides, exit status
[WARN] [1653488379.766281, 904.979000]: Docking attempt failed
[WARN] [1653488379.772542, 904.985000]: Retry attempt: 1/1
[INFO] [1653488379.777406, 904.989000]: Status is not retryable
[WARN] [1653488379.779995, 904.992000]: Unable to retry
[WARN] [1653488379.785579, 904.996000]: Status: [IDLE] | Dock Action failed to run

It seems that only one marker is detected when it is trying to enter the "steer_dock" state, that's why the action failed?
Can you check with the /fiducial_images image topic viz if the markers are still observable py the aruco_detect. We have encountered that tag detection is prone to the lighting condition of the local environment. Also, it seems that the robot isnt going into "parralel correction" state, you can also try to reduce the threshold of  max_parallel_offset param.

Hi youliangtan,
thanks for your response
at the first time the robot is parallel to the 3 tags and it can detect all of them. I don't understand why it tries to turn left instead of going straight to the charging station. when it tries to turn left it loses the detection of the tags so it can't go to the station.
actually  max_parallel_offset: 0.16.

In the predock state, the current logic is that the robot will align itself vertically to the detected side markers. This realignment action will happen even if only 1 marker is detected.
I think there's some misinformation in the printout of this. This line get printed even if a marker is detecting 1 marker (do_single_side_marker_rotate()). So I believe that from the log that u have provided here, the robot is not able to detect fiducial_10. Do check the image view of the robot. All in all, the robot needs to detect both sides of the marker only to transition successfully to the next "steer_dock" state.

at first the robot detected the 2 markers ( i verified this with image view )so it started to rotate in its place then it could detect just the marker fiducial_10 then it followed the rotation until no marker was detected.
which parameters should i verify to ensure that the robot pass to steer_dock state.
i reduced  max_parallel_offset as you mentioned but there is no result .

Hi youliangtan
I resolved the problem of tags but the robot stop at 1m from fiducials tags and publish that he arrive to the docking station.

Nice that you resovled the tags problem. It will be nice if you can share what changes you made to solve this.
Related to the robot stopping 1m from the tags, this is because the robot lost detection of the tags, and transitioned to "last_mile" state. Thus, try to adjust this value, which let the robot relies on pure odometry to move towards the charger.

i changed these values in turtlebot3.yaml
stop_yaw_diff: 3     # radian
max_parallel_offset: 0.05

to_last_mile_dis: 0.3         # edge2edge distance where transition to LM
to_last_mile_tol: 0.1         # transition tolerance from SD to LM
these parameters does not effect to the stopping distance from tags?

to_last_mile_dis and to_last_mile_tol that you mentioned are the condition which the state transition from "steer_dock" to "last_mile". Changing these 2 params help to ensure that the robot will move with odometry during the very last mile of docking (without relying of visual markers).

@youliangtan the same thing the robot stay 0.6 m (from tags when i try to set these value)
max_last_mile_odom: 0.05 #0.2
stop_distance: 0.08 #0.08        # edge2edge distance to stop from charger
when i try to increase the value max_last_mile_odom to 0.25 the robot still go forward until he losed the connection with tags and print IDEL.

I think you might need to increase the value of max_last_mile_odom. This is the value which the robot will move based on odometry, after losing the visual marker detection. Unfortunately, this pkg requires some tuning to make it work on different robots. Hopefully, you can provide me with a simple example for me to replicate this issue. If this is indeed an issue, a fix might be needed.


Popup doesn't close after changeset is uploaded

Currently on the live site the Uploading changes to OpenStreetMap popup doesn't close itself after a changeset is uploaded. The changeset closes just fine on the server but the page needs reloaded to continue mapping.
Windows 10 64-bit
Vivaldi 2.3.1440.61

I am having the same issue with Chrome on Linux.  Here is an image of what iD looks like 5 minutes after uploading has finished:

This is 100% reproducible in Safari from my tests. The issue is with fetching the community index list. When you see the "You just edited OpenStreetMap!" sidebar in the background then your changes were successfully uploaded. The box will never disappear so until we fix this please just reload the page to keep editing!


view: 🔭 exhaustive pattern in Planner reset

see #3955.
Planner::plan_with_spendable_and_votable_notes() resets all of its fields, except for fee_tier.
this does not address that "bug", per conversation in discord:

kate [penumbra] i noticed that Planner::plan_with_spendable_and_votable_notes() does not clear the fee tier field. i believe that is a dormant bug. would a PR to fix that be welcomed?
hdevalence [penumbra] i'd hold off on it, i would prefer to throw that code away
hdevalence [penumbra] there should be no clearing of fields at all, for instance

to prevent other such bugs creeping up in the meantime, we add an exhaustive pattern destructuring the Planner, so that future fields added to the planner must be contended with.

https://github.com/penumbra-zone/penumbra/issues/3955#issuecomment-1979699326


Help people who forget to change the status to needs review

Here's a test to show how this works: http://jsfiddle.net/ssstgm16/2/
If a file is in the browse file field or uploaded and is a patch or diff file, this will check the status is not NR or RTBC and if so put up a JS confirm to help catch the user into setting the status before submitting.

Dreditor is being decommissioned due to the lack of availability of current maintainers and the fact that Drupal.org is becoming more feature rich at a much faster rate than Dreditor itself now.
This decision was also partially made due to the amount of work (complete rewrite) it would take to create an "official" Firefox extension (#256).
If you already have the extension installed, it should continue working for a while until, inevitably,  the markup on Drupal.org changes. It will no longer be updated.
We understand that many of you love this extension, as do we. This isn't the end, in fact, it's a much brighter beginning. All the things we love about Dreditor can, and should, be moved into Drupal.org natively.
For more information, please follow: https://www.drupal.org/node/2779729

This issue was moved to dreditor/dreditor#59


Does stree work on Windows computer?

Looks stree is very interesting project for me. I try to use on Windows 7 using git bash (1.9.5), but get an error messages to run the demo command (https://medium.com/@porteneuve/mastering-git-subtrees-943d29a798ec)
git stree add demo -P vendor/plugins/demo ../remotes/plugin master

C:\Program Files (x86)\Git/libexec/git-core\git-stree: line 32: conditional binary operator expected
C:\Program Files (x86)\Git/libexec/git-core\git-stree: line 32: syntax error near `=~'
C:\Program Files (x86)\Git/libexec/git-core\git-stree: line 32: `tty -s <&1 && [[ "$TERM" =~ color ]] &&      is_tty=true || is_tty=false'

Hey there,
Yes, this is a known issue: #6 — Will do soon!


Match tests added

Seems the internet is down :) https://downdetector.com/status/google

http://uncyclopedia.wikia.com/wiki/AAAAAAAAA!

HAHAHAAAAAAA!

restarted...

Epic: first time two Javaslang PRs build simultaneously :)


Tauri V2: Migration thread

Tauri v2 has been released(in beta currently). It would be nice to start thinking of migrating to it. There are a lot of API changes though, and the entire rust backend will need to be almost rewritten

update: revisited v2 branch of tauri again, still doesn't seem to be ready for usage. Will revisit later - probably when it leaves beta branch. Most likely, the entire backend will need a thorough refresh - but with a stable API it should be a breeze

tauri v2 RC is here - https://v2.tauri.app/blog/tauri-2-0-0-release-candidate/. The stable RC should be coming at the end of this month. the work can be resumed 🎉

tauri v2 without breaking changes has finally been released 4 days ago - https://v2.tauri.app/blog/tauri-20/ - will re-start porting the app in the coming weeks.


How to save model weights after transfer learning

How to save the adjusted model after updating the model weight using domain adaptation or transfer learning? For example, I used the TrAdaBoostR2 method. But when I save the model, whether it is model.save or model.save_ weights have no effect.

Hi @9527-ly,
Thank you for your interest in the adapt library.
TrAdaBoostR2 has no save_weights method. However, you may succeed to pickle the object using joblib:
import joblib

joblib.dump(model, "model.pkl")

loaded_model = joblib.load("model.pkl")

If for some reason it doesn't work, you can always save the weights of every estimator in TrAdaBoostR2:

for i in range(model.n_estimators):
    model.estimators_[i].save_weights("model_%i.hdf5"%i)

Please, do not hesitate if you have further questions.
Best,

Thank you for your answer. I have succeeded to pickle the object using joblib


RFC: Background object cache maintenance job

The (shared) object cache is growing to be large. This includes:

Number of loose objects.
Number of pack-files.
Size of pack-files.

There are some things we can do to improve this situation, with varying levels of difficulty and benefit. Here are the four items I believe will have a strong impact on our data size and performance:
I. Delete loose objects that exist in pack-files.
II. Place loose objects into pack-files.
III. Repack some prefetch packs into larger packs.
IV. Delete packs with no objects referenced by the multi-pack-index.
These are numbered by order of implementation, but they work together in an interesting way. If we place loose objects into pack-files (I), then the next time we delete loose objects (II) we will delete those objects that we just put into a pack-file. If we repack some prefetch packs into larger packs (III), the multi-pack-index will use the "newer" pack for all objects in those packs, and we can delete them (IV).
In order to perform this maintenance without disrupting users, we can actually do these interactions in two independent jobs, with their sub-actions in a particular order:
A. Loose Object Maintenance

Delete loose objects that exist in pack-files.
Place loose objects into pack-files.

B. Prefetch Pack Maintenance

Delete packs with no objects referenced by the multi-pack-index.
Repack some prefetch packs into larger packs.

Care must be taken to ensure that Git processes that initialized their object store before the maintenance   still succeed. Thus, we should never delete anything that those processes could be reading from. This is why we delete loose objects only if they are referenced by pack-files. This is why we delete prefetch packs only if they have no objects referenced by the multi-pack-index. Since a second run of these jobs could delete objects that were repacked in the previous run, we need to space our job runs such that no Git processes span both runs.
Loose Object Job
The loose object job can save users who have many loose object downloads.  Older enlistments have many loose commits and trees due to the behavior of git fetch before the --no-show-forced-updates option.
 This will use the multi-pack-index to test containment, so is only bound by the file-deletion operations.
Pack loose objects that do not appear in a pack-file
We can use git pack-objects -q <object-dir>/pack/loose-<timestamp> to create a pack-file filled with a list of loose objects. We can pass the list of loose objects to this command through stdin. We can choose to batch this operation by limiting the number of objects we pack at a time, if we want. We may also want to skip this step if the number of loose objects is low (< 1,000? < 10,000?)
By listing a timestamp in the pack-file name, we can store the "most-recent" time this job was run.
Note: While we are thinking about packing loose objects, we can consider deleting some loose blobs. It is easier to delete loose blobs than packed blobs. If we want to delete "stale" blobs, then we should add a step to this job.
Prefetch Pack Job
We store a list of prefetch packs as <object-dir>/pack/prefetch-<timestamp>-<sha>.[pack|idx]. These are downloaded at a regular cadence by the post-fetch job, and could have a new pack-file for every hour. In addition to these prefetch packs, some daily prefetch packs will appear and occasionally a very large prefetch pack covering everything more than 30 days old. This large prefetch pack appears on initial clone and sometimes again if a user does not prefetch for 30 days (this occurs less often now that background prefetch exists).
Delete pack-files with no objects referenced by the multi-pack-index
The multi-pack-index stores a single copy of every packed object. When an object appears in multiple pack-files covered by the multi-pack-index, we select the pack-file with the most-recent modified time as reported by the filesystem. For example, if someone downloads a second large prefetch pack when prefetching for the first time in 31 days, then that large prefetch pack is referenced and all older prefetch packs are "stale" and not referenced by the multi-pack-index.
We can extend the git multi-pack-index builtin to delete pack-files that have no objects referenced by the multi-pack-index. This should happen at the same time as we update the multi-pack-index since we don't want to have this list of pack-files in the multi-pack-index if they don't exist. We also don't want to remove them from the multi-pack-index while the pack-files still exist, because then a Git process will start referencing those pack-files in the packed_git linked list.
Repack the prefetch pack-files
We can reduce the number of prefetch packs and improve file locality by combining prefetch packs into new "repacked" pack-files. Use git show-index <object-dir>/pack/prefetch-<timestamp[i]>-<hash[i]>.idx to list the objects in the prefetch packs and pipe that output to git pack-objects -q <object-dir>/pack/repacked-<timestamp>.
We should refrain from packing the most-recent prefetch packs, as we don't want to get in the way of the prefetch machinery for determining the most-recent timestamp. We could, however, start repacking the prefetch packs from oldest-to-newest, stopping in the most-recent week, or when the prefetch packs are large enough to satisfy a batch. Some stopping conditions could include "pack-file size is above 1GB".
Running the Jobs
We already have a post-fetch job that is automatically queued in the background. We should rename this job to be a "maintenance" job that could trigger several actions. The maintenance job should run every 15 minutes (as the post-fetch job currently does) but then it should check each "sub-job" for a timing condition to run again.
For example:

The post-fetch job doesn't trigger if the most-recent prefetch pack timestamp is within the last 70 minutes.
The loose object job doesn't trigger if the most-recent "loose" pack timestamp is within the last 24 hours.
The prefetch repack job doesn't trigger if the most-recent "repacked" pack timestamp is within the last 7 days or the most-recent "repacked" pack is modified within the last 24 hours.

Each job that may delete data should take a file-based lock on the object cache. This could be a "maintenance lock".
In addition to this background activity, we can enable manually running these jobs as a verb, say gvfs maintenance.
Secondary Concerns
We will want to track some data improvements using logging. We could report the number of loose objects before and after the loose object job, for instance. We could report the number and total size of pack-files, possibly grouped by the prefix; we expect pack-files with prefixes including "pack", "prefetch", "loose", and "repacked".

Mentioning @jrbriggs, @sanoursa, @kewillford, @wilbaker, @jamill, @nickgra, @alameenshah, @jeschu1, @mjcheetham for feedback. Thanks!

Please use this PR review for comments.

Use #492 for further design discussion. The problems this design tries to solve are covered in #85.

Reopening, as this is rewritten to be about the new cleanup steps to add in the future. @jeschu1 is working on the loose object cleanup step next, I believe.


prometheus-blackbox-exporter: 0.22.0 -> 0.23.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/prometheus/blackbox_exporter/releases.
meta.description for prometheus-blackbox-exporter is: "Blackbox probing of endpoints over HTTP, HTTPS, DNS, TCP and ICMP"
meta.homepage for prometheus-blackbox-exporter is: "https://github.com/prometheus/blackbox_exporter"
meta.changelog for prometheus-blackbox-exporter is: ""
Updates performed

Golang update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 0.23.0 with grep in /nix/store/kyy3y0w971jsn1v71nnd86ydvhmqksa6-blackbox_exporter-0.23.0
found 0.23.0 in filename of file in /nix/store/kyy3y0w971jsn1v71nnd86ydvhmqksa6-blackbox_exporter-0.23.0

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
prometheus-blackbox-exporter

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/kyy3y0w971jsn1v71nnd86ydvhmqksa6-blackbox_exporter-0.23.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A prometheus-blackbox-exporter https://github.com/r-ryantm/nixpkgs/archive/55ef419a6e2b4536549e6115d66a11f611674cbc.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/kyy3y0w971jsn1v71nnd86ydvhmqksa6-blackbox_exporter-0.23.0
ls -la /nix/store/kyy3y0w971jsn1v71nnd86ydvhmqksa6-blackbox_exporter-0.23.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

prometheus-blackbox-exporter

Maintainer pings
cc @globin @fpletz @WilliButz @Frostman @Ma27 for testing.

Result of nixpkgs-review pr 204507 run on x86_64-linux 1

1 package built:

prometheus-blackbox-exporter


tr: how should non-UTF-8 input be handled?

Introduction
The current tr implementation assumes that the data fed into standard input is valid UTF-8.
Most implementations of tr work on a byte by byte basis, which allows non-UTF-8 data to be handled.
BusyBox, GNU Core Utilities, and uutils' coreutils:
❯ printf '\xFF\n' | /usr/bin/tr '\377' 'A'
A

Multibyte UTF-8 data is obviously not handled correctly:
❯ printf '%s\n' 'ᛆᚠᛏᚢᛆᛘᚢᚦᛌᛏᚭᚿᛏᛆᚱᚢᚿᛆᛧᚦᛆᛧ' | /usr/bin/tr -d 'ᛆᚠ'
����������������

bsdutils (port of FreeBSD tools to Linux) handles multibyte UTF-8 input correctly:
❯ printf '%s\n' 'ᛆᚠᛏᚢᛆᛘᚢᚦᛌᛏᚭᚿᛏᛆᚱᚢᚿᛆᛧᚦᛆᛧ' | /usr/ucb/tr -d 'ᛆᚠ'
ᛏᚢᛘᚢᚦᛌᛏᚭᚿᛏᚱᚢᚿᛧᚦᛧ

But that means it can't handle non-Unicode input:
❯ printf '\xFF\n' | /usr/ucb/tr '\377' 'A'
tr: Invalid or incomplete multibyte or wide character

The posixutils implementation is currently similar to the bsdutils implementation, but the error message needs to be changed:
❯ printf '\xFF\n' | ./target/release/tr '\377' 'A'
thread 'main' panicked at text/./tr.rs:1200:25:
assertion failed: leftover_bytes == 0_usize
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Options
In no particular order:

Keep the current implementation (assume UTF-8) input, but just fix the error message when tr is fed non-UTF-8 data

Pros: supports some use cases that the widely used implementations do not
Cons: users may expect to be able to use tr to perform arbitrary byte transformations (not thinking of their input as text)

Mirror most implementations and just process input on a byte by byte basis.

Pros: better performance, simpler code, behaves like the most widely used implementations
Cons: can't perform transformations of multibyte text

Support both byte by byte processing and UTF-8 input, either by starting in UTF-8 mode and then switching to byte by byte mode when non-UTF-8 data is detected, or by providing additional/non-standard arguments

Pros: supports basically every use case
Cons: code complexity, deviation from existing implementations

I don't think there's one clear right option here, so please chime in if you have any suggestions or thoughts.

There is a core directive, and a practical directive:
Core directive:  Be aggressive in pushing UTF-8 support, as an upgrade to the old utilities.  This is our fresh take.
Practical directive:  Do legacy scripts in the BSD and Linux world require byte-for-byte?  More practically, does our tr break lots of in-the-field existing scripts?  If yes, we should support B4B mode because posixutils should be useful in the real world and not break lots of existing scripts.
That argues for either UTF8 only (number 1) or UTF8 + B4B (number 3).

Thanks, that helps. Did some quick Googling:
Here's someone being advised to use tr to remove non-ASCII characters from file with a mixture of ASCII and arbitrary/unknown bytes:
https://stackoverflow.com/questions/15133607/how-to-remove-non-ascii-chars-using-sed/15134539#15134539
Similar to above:
https://stackoverflow.com/questions/3337936/remove-non-ascii-characters-from-csv/47918586#47918586
Considering this is just one use case out of many that one could think of that would require binary/non-UTF-8 handling, and how long tr has been around, I would guess that only supporting UTF-8 would cause significant breakage.
Please let me know if you're okay with number 3, and if so, if you think auto-fallback should be used, or command-line arguments. My suggestion would probably be to do both (something like --force-binary, --force-utf-8), so the user has recourse in case something goes wrong with the UTF-8 to binary auto-fallback. This might sound like a lot, but I don't think it will make the code much more complicated.

The wc utility has a bytes mode and a chars mode...

Closed by #354


Formatting issues for field names/types

See https://github.com/meteor/docs/issues/53

See also https://github.com/meteor/docs/issues/41 and https://github.com/meteor/docs/issues/57

Before:

After:

Thanks for sorting this out @lorensr


Feedback

Things related to feedback/Comments

https://twitter.com/mavrck/status/1178088479798026240?s=19
A lot of ideas and pros and cons are collected here


#102 better-chef-rundeck support

this is not ready yet, just putting progress up here...

this is being handled elsewhere


i18n(fr): Updating framework-components translation

What kind of changes does this PR include?

[x] New or updated content
[x] Translated content

Description

What does this PR change? Updates content from src/pages/fr/core-concepts/framework-components.md to reflect latest changes

TYSM @delucis for taking your time on this ! All the suggestions make sense, now some sentence make actually sense 😆

Nice work @Chrissdroid! There is a small change from #862 that I'm not sure if it impacts your translation, just pointing this out if you want to add it as part of this PR. Let me know when this is good to merge! 🚀

Nice work @Chrissdroid! There is a small change from #862 that I'm not sure if it impacts your translation, just pointing this out if you want to add it as part of this PR. Let me know when this is good to merge! 🚀

Sorry for the delay on this @Yan-Thomas ! Thanks for notifying me on this ! 💯

LGTM. Thanks @Chrissdroid!


PDI-19907 - Moved Dimension Lookup into Plugin

:white_check_mark: Build finished in 1h 1s
Build command:
mvn clean verify -B -e -Daudit -amd -pl \
engine,plugins,plugins/db,plugins/db/dimlookup,plugins/db/dimlookup/assemblies,plugins/db/dimlookup/assemblies/plugin,plugins/db/dimlookup/core,ui

:ok_hand: All tests passed!
Tests run: 5327, Failures: 0, Skipped: 14    Test Results

:information_source:  This is an automatic message

:x: Build failed in 1m 55s
Build command:
mvn clean verify -B -e -Daudit -amd -Djs.no.sandbox

:exclamation: No tests found!
:information_source:  This is an automatic message


Global announcement admin should get all announcements

Description
Changes global announcements admin get API to return all global announcements, rather than just the active ones.
Motivation and Context
The sysadmin needs to be able to manage all announces, including those that have expired and those that are upcoming.
How Has This Been Tested?

Updated existing test to ensure that all global announcements are returned to the admin. Rant test. Test failed.
Fixed response. Ran test. Test passed.

Checklist

N/A - I have updated the documentation accordingly.
[ ] All new and existing tests passed.

I'm rerunning the checks.  There may be a problem with the tests, FYI.

Actually on closer examination it looks like there's a problem with one of your tests:
______________________ TestAnnouncementService.test_post _______________________
self = <tests.api.admin.controller.test_announcement_service.TestAnnouncementService object at 0x7f853d8b0610>
  def test_post(self):
      with self.request_context_with_admin("/", method="POST") as ctx:
          ctx.request.form = MultiDict([("announcements", json.dumps([self.active]))])
          response = AnnouncementSettings(self.manager).process_many()

      announcements = Announcements.for_all(self._db)

  assert len(announcements.announcements) == 2

E       assert 1 == 2
E        +  where 1 = len([<api.announcements.Announcement object at 0x7f8523ab3bb0>])
E        +    where [<api.announcements.Announcement object at 0x7f8523ab3bb0>] = <api.announcements.Announcements object at 0x7f8523a8b2b0>.announcements
tests/api/admin/controller/test_announcement_service.py:38: AssertionError

Actually on closer examination it looks like there's a problem with one of your tests:

Thanks! It was an inadvertent change. I meant to fix checks only in the test_get test, since the get method was the only place I saw the problem.


Send value from UI to Server

First, I have to say I'm a real noob in Rust
I want to send additional value to my Linux board when clicking the submit button.
So, I added input form in index.html easily but I wonder how to send it to the board
Actually, my board will do something with the delivered value after connecting with WiFi
Could you please help me about that?
Plus, how can I build the source?
Fortunately, I know how to use Docker
My Linux board's architecture is ARM 64, and the OS is Debian 9

We have an example of doing that through Python: https://github.com/balena-io-playground/wifi-connect-api/

Going to add some documentation now at the README of that repo.

I added additional information to the README. Please let me know if you have any questions.

@majorz
Thank you so much! I'll try

@majorz
I've tried several times but it doesn't work
It seems working fine but my phones and laptop could not find the AP(WiFi-connect)

Serving Flask app "web-app" (lazy loading)
Environment: production
WARNING: Do not use the development server in a production environment.
Use a production WSGI server instead.
Debug mode: off
Running on http://<IP_ADDRESS>:80/ (Press CTRL+C to quit)

I did docker build/run as below
docker build --tag wifi-connect .
docker run -p 80:80 -p 45454:45454 wifi-connect
And, I had to change some words in Dockerfile.template as below
Because, these variables cannot found
%%RESIN_MACHINE_NAME%%
%%RESIN_ARCH%%
FROM balenalib/generic-aarch64-python:3.7.2-20190327
RUN curl -Ls https://github.com/resin-io/resin-wifi-connect/releases/download/$WW
IFI_CONNECT_VERSION/wifi-connect-$WIFI_CONNECT_VERSION-linux-aarch64.tar.gz | taa
r -xvz -C /tmp/download

@theruin0000 Docker will need --cap-add=NET_ADMIN (or --privileged) and --network=host, as those are required by dnsmasq.
Additionally you will need to mount the DBus socket in Docker. I need to look up how that is done on non-balenaOS environment.
Do you actually need Docker? Probably the best way is to download wifi-connect and put it in /usr/local/sbin on the OS directly. And follow the other steps from the Dockerfile.template but apply them on the OS. What do you think?

Do you mean following Dockerfile.template step by step manually?
Okay, I'll try :)

Yes, it is only a few steps like download and copy wifi-connect, clone the repo, install Python requirements, and run the application.

@majorz
I just found it works well :)
So thanks for your help!


RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

The full error:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [3, 4]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

I've looked through all the source code, and the only obvious in-place operations are in losses.py:
https://github.com/tristandb/EfficientDet-PyTorch/blob/b86f3661c9167ed9394bdfd430ea4673ad5177c7/losses.py#L75
https://github.com/tristandb/EfficientDet-PyTorch/blob/b86f3661c9167ed9394bdfd430ea4673ad5177c7/losses.py#L83-L84
Not sure how to solve it yet, but will post back here when I do.

I also found these two lines:
https://github.com/tristandb/EfficientDet-PyTorch/blob/b86f3661c9167ed9394bdfd430ea4673ad5177c7/utils.py#L17
https://github.com/tristandb/EfficientDet-PyTorch/blob/b86f3661c9167ed9394bdfd430ea4673ad5177c7/utils.py#L54
but when I change them to inplace=False, the error is still there.

Turns out that, after narrowing down the problem to the BiFPN code, the fix is to change these lines:
https://github.com/tristandb/EfficientDet-PyTorch/blob/b86f3661c9167ed9394bdfd430ea4673ad5177c7/bifpn.py#L73-L77
To look like this:
        # Calculate Top-Down Pathway
        w1 = self.w1_relu(self.w1)
        w1 = w1.clone() / (torch.sum(w1, dim=0) + self.epsilon)
        w2 = self.w2_relu(self.w2)
        w2 = w2.clone() / (torch.sum(w2, dim=0) + self.epsilon)

@tristandb this still needs to be fixed in master. I won't close this issue yet.

Can we use w1 directly instead of w1.clone()?


publicapi: 400 body returned from API server: client's signature is invalid

building from source

using the binary

if you’re building from source, you also need to be communicating to a local devstack server - otherwise you’ll be out of so in a new shell do, ./bacalhau devstack .and in the first shell, paste in the shell variables


Always create RitualSpellcasting instances

(and defer retrieval of rituals for their collections)

If I recall, this deferral doesn't really lead to anything, since ActorSpellcasting retrieves the collections immediately. But I suppose that may change if collection retrieval is deferred.