Fix: "TypeError: Cannot read properties of undefined (reading 'length')" in on('virtualUpdate') by checking if swiper.slides is set

Hello
I'm using the react components with in a fairly complex component using the virtual settings. To force a full rebuild of swiper (e.g. while toggling between fullscreen and not fullscreen view) I update the key param.
By doing so I experienced that an unhandled exception in the requestAnimationFrame enqueued by on('virtualUpdate').
Chrome: TypeError: Cannot read properties of undefined (reading 'length')
Firefor: TypeError: swiper.slides is undefined
I suppose this happens because the requested animation frame is being processed after or while the change of the key reset the swiper object.
I've only added a check if swiper.slides is set at all and could remove the error in my code.
Thank you!

Merged, thanks!


flickering timer

The timer when another player is answering or when you are answering can start flickering.

issue to do server sending out more than one "paused" response over timer


Street of the Sisters

Search for "King’s Landing" (https://thronesdb.com/find?q=k%3A"King’s+Landing"&sort=faction&view=scan) do not have Street of the Sisters (https://thronesdb.com/card/02018)

Try https://thronesdb.com/find?q=k%3A"King's+Landing"&sort=faction&view=spoiler


flow v0.57.x don't shows no errors if use module.system.node.resolve_dirname=src

I need help!
flow don't show any errors, if write module.system.node.resolve_dirname=src in .flowconfig
https://github.com/gin0606/flow-v0.57.1-trouble
Please see this repository, that can see my trouble.
Thanks!
Maybe related to https://github.com/flowtype/flow-bin/issues/91 and https://github.com/facebook/flow/issues/5105

well this explains my issue that i reported as well... flow team doesnt respond to anyone so who knows if it will get fixed but we can hope

Still an issue with 0.58.0

I'd encountered the same problem, and dug down into a code of the Flow, then I think that I found what the reason it caused.
First, in version 0.57 of Flow, the feature as described below added

New Features:
Flow will now only check the files in node_modules/ which are direct or transitive dependencies of the non-node_modules code.

Also, the document explains about setting module.system.node.resolve_dirname as below.

module.system.node.resolve_dirname (string)
By default, Flow will look in directories named node_modules for node modules. You can configure this behavior with this option.

In other words, adding a setting to resolve_dirname means that a directory added will not be checked as well as node_modules unless imported from the file where not in the directory set to resolve_dirname and not set to the ignore option.
As far as the document explained, I think this behavior may be the intended of the Flow team.
But it will confuse some users, including me.
My workaround is to import files where Flow check lazily, from files where Flow check immediately
For example, in case of @gin0606 https://github.com/gin0606/flow-v0.57.1-trouble, we will be able to force type checking by placing the file like below in the root directory of the repository.
// ./src/index.js
// @flow
import bar from 'foo/bar';

let num: number = 1;
let numerror: number = 'string';
let string: string = bar();

console.log(num);
console.log(numerror);
console.log(string);
+
+ export default () => {};

// In root of repository ./root.js
+ import fn from "./src"

But the workaround seems not absolutely ideally.
My personal opinion is that it should be checked lazily node_modules directory only.
Because some users use resolve_dirname option conjunction with resolve.modules option of the Webpack.
If we need PR to make such corrections, I will be honored to make it.
However, for us to find an ideal solution about this, I would like to hear the opinion of the Flow team on this Issue especially @gabelevi 's one.

I actually implemented a workaround for this problem. It doesn't keep the exact same behavior but you get your errors back in the case described by the OP:
Simply change:
module.system.node.resolve_dirname=src

to
module.system.node.resolve_dirname=./src

Thanks @kogai !
For our project we've just replaced module.system.node.resolve_dirname=src with several module.name_mapper, like this:
module.name_mapper='^desktop/\(.*\)$' -> '<PROJECT_ROOT>/src/desktop/\1'
And it seems that it fixed the issue.

On the project I am using now the relative url definitely fixes the issue.

I'd like to echo the question from @kogai.
We are using Flow in a fairly large project, with the module.system.node.resolve_dirname rule pointing to our local source code.
Maintaining a list of individual name_mapper rules is going to be quite unwieldy and error prone.  Converting every import statement to a relative import will also be difficult at this stage, and seems likely to increase the fragility of our codebase.
It would be great to hear the flow team's thoughts on this issue, and investigate possible alternatives.

Experiencing the same issue. But here is the weird thing: on OSX it works perfectly fine. On our CI (Travis, Ubuntu 14.04.5 LTS) it fails to output any errors.
Unfortunately the workaround of a list of name_mapper rules doesn't work. Maybe a difference with others is that we run flow check with the --all flag.

@qamil I had the same issue just yesterday, and although I know that it probably won't help everyone, I made CI checking work by changing module.system.node.resolve_dirname=src to module.system.node.resolve_dirname=./src

Ok, my bad. It did work for a while yesterday, but it's not working anymore...

I don't have resolve_dirname configured (in fact, there's nothing in my flowconfig file) and yet I have this problem - flow always prints "No errors!" no matter what I do (I do have the /* @flow */ pragma at the top of the file).
But in my case it's even weirder, I'm using sublime with SublimeLinter-flow, and it, somehow, works properly.

Any idea why the ./ trick works? Thanks to @bradennapier I've now got this:
[options]
module.system.node.resolve_dirname=node_modules
module.system.node.resolve_dirname=./app

...and it works great – Flow doesn't check non-depended files within node_modules, but does check everything in app.
But I'm concerned this might be an unsupported hack, that it might get 'fixed' in a future Flow version and stop working. Is the ./ trick an approved way to opt-out of the 0.57.0 new feature, or is there a better method that is sanctioned by the Flow team?

@callumlocke It seems like it works for me as well (with Flow 0.66.0), thanks!

@alexeyraspopov I can't see an answer to my question there. Can you be more specific?

@callumlocke, do not use resolve_dirname for your code. Use name_mapper instead.

@alexeyraspopov can you point to where a Flow team member has said this is the better course of action? Or some official docs saying this?
My app has a lot of subdirs. Using several name_mapper lines (one for each subdir under app) is more complex, and a bit harder to maintain (need to remember to update the flow config whenever our directory structure changes). So before doing that, I want something official.

@alexeyraspopov you are right, usage of module.name_mapper is the correct solution. It's a pity that it is not clear from the official documentation especially when module.system.node.resolve_dirname was commonly used for aliases.
@callumlocke for every folder under src add the following line to [options] section of your .flowcong file:
module.name_mapper='^my_folder/\(.*\)$' -> '<PROJECT_ROOT>/src/my_folder/\1'
BTW Now I think this issue can be closed... :)

can you point to where a Flow team member has said this is the better course of action? Or some official docs saying this?

This is why my issue exists. Docs need improvements to emphasize this part.

@callumlocke I agree that it is harder to maintain but the official documentation (https://flow.org/en/docs/config/options/#toc-module-system-node-resolve-dirname-string) says that module.system.node.resolve_dirname should be used for specifying of node modules location. Using it for aliases of our modules was kind of hack I think.

@JakubRimal @alexeyraspopov OK, I have PR'd a docs update. Please go and upvote it if you agree: https://github.com/facebook/flow/pull/5850

This issue is also being tracked at https://github.com/yahoo/react-intl/issues/1086

module.system.node.resolve_dirname should be used for specifying of node modules location. Using it for aliases of our modules was kind of hack I think.

Right, it's bad that this hack is the only sane way to make flow work as expected right now.

Resolving modules from directory is a hack itself. It's not expected by browser or node.

And yet, it's intensely useful and I wouldn't do a project without it.
On Mar 5, 2018 11:59 PM, "Bogdan Chadkin"<EMAIL_ADDRESS>wrote:

Resolving modules from directory is a hack itself. It's not expected by
browser or node.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/facebook/flow/issues/5180#issuecomment-370673006, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABJFP7TolgMcBXkdOrWKLUfMRBXqmA9vks5tbiW-gaJpZM4QFbXC
.

Just a question to be sure to understand:
If I have 60 components in src/components/ like so:
src/
├── components/
│   ├── Button/
│   ├── Loader/
│   ├── RainMaker/
│   ├── .../

Do I have to write 60 name_mapper lines like:
module.name_mapper='^Button\/\(.*\)$' -> '<PROJECT_ROOT>/src/components/Button\1'
In order to import a component like:
import Button from 'Button';
??
It is possible to do like so?
import Button from 'components/Button'
?
Thanks in advance

Just one if you're doing components/. Yes, if you wanted to do all of them with no prefix, you'd have to write a mapper for every single directory. I wouldn't recommend that pattern anyway.
module.name_mapper='^components\/\(.*\)$' -> '<PROJECT_ROOT>/src/components/\1'

@stasgavrylov 's workaround (https://github.com/facebook/flow/issues/5180#issuecomment-344226227) works for us. But we would really like this issue to be fixed officially by Flow.

The module.system.node.resolve_dirname=./src doesn't work on 0.83.0

I just started encountering the same issue on flow 0.119.1
no errors are shown while there are several in the code. Changing a file will introduce some errors but the behavior is completely erratic
Using:
module.system.node.resolve_dirname=./src
instead of:
module.system.node.resolve_dirname=src
makes flow report every import for a file inside my project (under src) as unable to resolve.
Did anyone find a solution that doesnt include writing custom regex mapper for each folder in my src root?
Having to maintain a matching list of folders seems like a terrible way to go about keeping my code safe...


Chore: format using rome

:tada: This PR is included in version 3.7.0-beta.7 :tada:
The release is available on:

GitHub release
v3.7.0-beta.7

Your semantic-release bot :package::rocket:


Generate title pages without the YAML file?

I'm using SILE on the back end of a flow that can generate HTML and PDFs.  I'd like to be able to use the resilient.book title and half-title pages, but these don't seem to be documented for use except via a YAML.  It looks as if these come from the resilient.bookmatters packages, but the commands there aren't documented.

As of yet, package resilient.bookmatters is considered an internal component, with unstable commands (hence undocumented and subject to change). It's just the bare support for some of the things needed by the "master document" (expressed in YAML indeed).
What's the rationale for not using a master document? (It would be possible to parse it even for HTML generation, and even without using SILE at all here -- I have a yet unreleased (and uncompleted / poorly coded) experimenal workflow for HTML/ePub where I just do this.)

and uncompleted / poorly coded

Just to give more context: One of the points left aside for the future in that dirty experiment was a transpiler for the resilient style file to CSS. The idea being that most of my usual content being in Djot/Markdown, I would have a SILE path for nice PDF generation, and a (mostly) non-SILE one for direct HTML conversion.

I have a flow that parses some semantic markup and produces a document tree that I can then run transforms over.  For HTML output, the resulting tree is modified until it's valid HTML and emitted (and split into multiple files, one per chapter), for PDF I'm generating an XML file that's consumable by SILE.  The transforms are different for the two kinds of output, but the document structure is the same.
I could generate an additional file for the SILE output, but the internal document model maps trivially to XML and so it's easy to just generate more XML commands inline, whereas generating YAML requires creating a new kind of serialiser to some internal state and that's more work.
This is mostly a C++/Lua reimplementation of something I built for ePub + LaTeX typesetting of my third book after seeing how badly the publisher's PDF->ePub flow mangled my second book.  I really want four outputs:

HTML for online reading.
ePub for eBook readers.
Online PDF (colours, cross references as links)
Print PDF (black and white, cross references as page numbers)

The first two are small tweaks of each other, as are the last two.  They're generated from the same semantic markup, but in the ePub / HTML flows most of that is preserved as div and span classes, whereas in the SILE flow a lot of it gets lowered to presentation markup.  I don't need resilient to be involved in the HTML flow at all - I don't want the same visual style for both because things that have fixed layout and things with variable layouts are quite different, but I can generate both from the same semantic markup.

Ah I see. Well, even if they were documented, the commands from bookmatters.resilient as of now cannot be expressed intuitively directly in SIL (XML) -- they have structured options (attributes).


fix(): FirebaseApp Class should not implement app.App

Checklist

Issue number for this PR: #2059
Docs included?: n/a
Test units included?: yes
In a clean directory, yarn install, yarn test run successfully? yes

Description
While giving us a bit more type safety in development, implementing app.App allows the Firebase JS SDK to break our builds in minor releases. Which has bit us more than once.
I do not believe this is worth keeping around any longer.
Code sample
No API change.

@davideast Heads up, the tests are stuck on the Firestore flake. I'm going to admin override & merge.


Update docs for helios-solo

Emphasize that docker needs to be started for helios-solo to work.
fixes #463

:+1:


No name is provided for close button: A11y_VS Code extension for API Center_Open API Documentation_Close_Name Role Value.

GitHub Tags
#A11yTCS; #A11ySev2; #Win32; #WCAG4.1.2; #GH_VSCodeextensionforAPICenter_Win32_Apr2024; #DesktopApp; #VS Code Extension for API Center; #Name Role Value; #AILimited; #A11yMAS; #External; #External:Swagger; #Thirdparty; eDAD:3P:Bug222927
Environment Details:
Application Name: VS Code Extension for API Center
OS: Windows 11 Enterprise 23H2 (OS build 22631.3296)
Repro Steps:

Create an API center in azure portal.
Open Visual studio Code.
Tab till API center extension in the left side panel and press enter.
Tab till definition under the Api name which you have created, and press enter.
The name which you have provided to the definition will appear for ex-"openapi4", right click on it.
Go to "Open API Documentation" using the down arrow and press enter.
Demo conference API screen will appear.
Tab till authorize button and press enter
Verify whether the name is provided for close button or not.

Actual Result:
No name is provided for close button.
Expected Result:
Name should be provided as a close for button.
User Impact:
Screen reader users will get the confuse if no name is provided for button.
Attachment:

@formulahendry Can you please provide us the standalone sample app or Code pen link to route this bug to external team.

IRT Bug logged: https://dev.azure.com/CSS-IRT/CSS IRT/_workitems/edit/222927

GitHub Tags:
#External;#External:SwaggerUI;#eDAD:3p:222927;ThirdParty;

Issue is not fixed in Version: 1.95.2 (user setup)

External bug: https://dev.azure.com/CSS-IRT/CSS IRT/_workitems/edit/222927


endless loop when trying to create roles with parent

Hi all.
I tried to setup ldap2pg to synchronize user and groups with LDAP.
There is a group within LDAP that contains all the users.
These group and the member users shall be created in Postgres, the privileges shall be assigned to the Postgres group.
Even when trying to run this heavily reduced ldap2pg.yml, the system goes into an endless loop and seems to build some memory structures until it gets killed by the oom_killer.
ldap2pg.yml
version: 6

postgres:
  schemas_query: |
    SELECT nspname FROM pg_catalog.pg_namespace
    WHERE nspname NOT LIKE 'pg_%' AND nspname <> 'information_schema'
    ORDER BY 1;

privileges:
  schemause:
  - default: global
    type: USAGE
    on: SCHEMAS

  reading:
  - schemause
  - default: global
    type: SELECT
    on: TABLES

rules:
- description: "Setup static roles and grants"
  roles:
  - name: pgro_dev_sys4
    options: NOLOGIN
  grant:
    - privilege: reading
      role: pgro_dev_sys4

- description: "Query LDAP to get members of readers"
  ldapsearch:
    base: "cn=groups,dc=sf-rz,dc=de"
    filter: "(cn=pgro_dev_sys4)"
  role:
    name: '{member.uid}'
    options:
      LOGIN: yes
      CONNECTION LIMIT: 5
    comment: "From LDAP group {dn}."
    parent: "{cn}"

Expectations

What you expected from ldap2pg ?

create the group role pgro_dev_sys4
put all the LDAP members of pgro_dev_sys4 into the Postgres group role

What ldap2pg did wrong ?
Went into an endless loop until killed by the oom_killer.

Verbose output of ldap2pg execution
time=2024-03-11T16:19:14.269+01:00 level=INFO msg="Starting ldap2pg" version=v6.0 runtime=go1.20.5 commit=023e6933
time=2024-03-11T16:19:14.270+01:00 level=INFO msg="Using YAML configuration file." path=/opt/schufa/pgsql/data/home/pginst_dev_sys4/ldap2pg/ldap2pg-mini.yml
time=2024-03-11T16:19:14.270+01:00 level=DEBUG msg="Loading YAML configuration."
..... (a lot of internal information, the personal userids show up from LDAP)
time=2024-03-11T16:19:14.461+01:00 level=DEBUG msg="Managing role." role=pgldap_dev_sys4
time=2024-03-11T16:19:14.461+01:00 level=DEBUG msg="Managing role." role=pgro_dev_sys4
time=2024-03-11T16:19:14.461+01:00 level=DEBUG msg="Managing role." role=public
time=2024-03-11T16:19:14.461+01:00 level=DEBUG msg="Managing role." role=schean
time=2024-03-11T16:19:14.461+01:00 level=WARN msg="Dry run. Postgres instance will be untouched."
Then no more output until the process is killed.
As soon as the line parent: "{cn}" is removed from the yaml, ldap2pg comes to an end but does not assign the personal users to the group role in PostgreSQL.

Hmm, odd. I failed to reproduce locally.
Could you share the Wants... log lines ?

Are you running agains Active Directory ? I'm wondering if this is an issue that dn is capitalized while {cn} is lowered. Which si fixed in master.

Closing stale isue. Please reopen with news. Thanks for reporting this.


[QUESTION] How to add images and data without data being below images and image cells expanded to image size?

[x ] Have read and understood: https://github.com/Maatwebsite/Laravel-Excel/blob/3.1/.github/SUPPORT.md
[x ] Checked if your Laravel Excel version is still supported: https://docs.laravel-excel.com/3.1/getting-started/support.html#supported-versions
[x ] Checked that your question isn't already asked before.
[x ] Filled in the entire issue template

Versions

PHP version: 7.3
Laravel version: 5.8
Package version: 3.1

Description
I am migrating from 2.1 to 3.1. In my export I would like to export images and data. I am exporting one image per row with a set height. The problem I am experiencing with using the collection and drawings methods in the class is that the collection data will start below the image data. With the images I am able to set the exact cell, with the collection data there is the startCell method, which seems not to work in addition to the drawings method...?
How can I:

Set Images without moving the data further down? I would like to start with the data in A2.
Set the A column width and height of each row in the sheet? I am have experimented with the macros similar to the fixedHeader, but was not able to achieve this.

Additional Information
My export class example
<?php

namespace App\Exports;

use Maatwebsite\Excel\Concerns\FromCollection;
use Maatwebsite\Excel\Concerns\WithEvents;
use Maatwebsite\Excel\Concerns\WithHeadings;
use Maatwebsite\Excel\Events\AfterSheet;
use Maatwebsite\Excel\Concerns\ShouldAutoSize;
use Maatwebsite\Excel\Concerns\WithDrawings;
use PhpOffice\PhpSpreadsheet\Worksheet\Drawing;
use Maatwebsite\Excel\Concerns\WithCustomStartCell;

class ReportExcel implements FromCollection, WithHeadings, WithEvents, ShouldAutoSize, WithDrawings, WithCustomStartCell {
    public function __construct($header,$data,$drawings = [])
    {
        $this->header = $header;
        $this->data = $data;
        $this->drawings = $drawings;
    }

    public function headings(): array
    {
        return $this->header;
    }

    public function registerEvents(): array
    {
        return [
            AfterSheet::class => function(AfterSheet $event) {
                $event->sheet->freezePane('A2', 'A2');
            },
        ];
    }

    public function collection()
    {
        return collect($this->data);
    }

    /*
    public function startCell()
    {
        return 'A2';
    }
    */

    public function drawings()
    {
        $drawings = [];

        foreach($this->drawings as $k=>$drawing_temp)
        {
            if($drawing_temp)
            {
                if($img_file = StoreImageUrlTmp($drawing_temp.'-h100'))
                {
                    $drawing = new Drawing();
                    $drawing->setName('image');
                    $drawing->setDescription('image');
                    $drawing->setPath($img_file);
                    $drawing->setHeight(70);
                    $drawing->setOffsetX(5);
                    $drawing->setOffsetY(5);
                    $drawing->setCoordinates('A'.($k+2));
                    $drawings[] = $drawing;
                }
            }
        }
        return $drawings;
    }
}

I was able to figure out column width and row height styling, see below. I am still searching on how to start the collection() content not after the drawings, but at the beginning of the sheet.
public function registerEvents(): array
    {
        $count = [
            count($data[0]), //column count
            count($this->drawings)?count($this->data):0 //row count, if there are images
        ];
        return [
            AfterSheet::class => function(AfterSheet $event) use($count) {
                //Freeze frist row
                $event->sheet->freezePane('A2', 'A2');

                //Set image column width; image column is always A
                $event->sheet->getColumnDimension('A')->setWidth(17);

                //Set auto width for the rest
                $columnindex = array('A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z',
                                     'AA', 'AB', 'AC', 'AD', 'AE', 'AF', 'AG', 'AH', 'AI', 'AJ', 'AK', 'AL', 'AM', 'AN', 'AO', 'AP', 'AQ', 'AR', 'AS', 'AT', 'AU', 'AV', 'AW', 'AX', 'AY', 'AZ',
                                     'BA', 'BB', 'BC', 'BD', 'BE', 'BF', 'BG', 'BH', 'BI', 'BJ', 'BK', 'BL', 'BM', 'BN', 'BO', 'BP', 'BQ', 'BR', 'BS', 'BT', 'BU', 'BV', 'BW', 'BX', 'BY', 'BZ');
                for($i=1;$i<$count[0];$i++) //iterate based on column count
                {
                    if($i>76) break;
                    $event->sheet->getColumnDimension($columnindex[$i])->setAutoSize(true);
                }

                //Set row height
                for($i=1;$i<$count[1];$i++) //iterate based on row count
                {
                    $event->sheet->getRowDimension($i+2)->setRowHeight(60);
                }
            },
        ];
    }

As I didn't find any way of repositioning the content I have shifted adding images now in the event method, which allows me to add the images after the other data from the collection has been added. It seem a little cumbersome to do all this in the event method, but it works. If there is a better way Id be happy to hear about it.
Full example of my reporting class:
<?php

namespace App\Exports;

use Maatwebsite\Excel\Concerns\FromCollection;
use Maatwebsite\Excel\Concerns\WithEvents;
use Maatwebsite\Excel\Concerns\WithHeadings;

use Maatwebsite\Excel\Events\AfterSheet;
use PhpOffice\PhpSpreadsheet\Worksheet\Drawing;

class ReportExcel implements FromCollection, WithHeadings, WithEvents {
    public function __construct($header,$data,$drawings = [])
    {
        $this->header = $header;
        $this->data = $data;
        $this->drawings = $drawings;
    }

    public function headings(): array
    {
        return $this->header;
    }

    public function registerEvents(): array
    {
        $count = [
            count($this->data[0]), //column count
            count($this->drawings)?count($this->data):0 //row count, if there are images
        ];
        $drawings = $this->drawings;

        return [
            AfterSheet::class => function(AfterSheet $event) use($count, $drawings) {
                //Freeze frist row
                $event->sheet->freezePane('A2', 'A2');

                //Set auto width for the rest
                $columnindex = array('A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z',
                                     'AA', 'AB', 'AC', 'AD', 'AE', 'AF', 'AG', 'AH', 'AI', 'AJ', 'AK', 'AL', 'AM', 'AN', 'AO', 'AP', 'AQ', 'AR', 'AS', 'AT', 'AU', 'AV', 'AW', 'AX', 'AY', 'AZ',
                                     'BA', 'BB', 'BC', 'BD', 'BE', 'BF', 'BG', 'BH', 'BI', 'BJ', 'BK', 'BL', 'BM', 'BN', 'BO', 'BP', 'BQ', 'BR', 'BS', 'BT', 'BU', 'BV', 'BW', 'BX', 'BY', 'BZ');
                for($i=0;$i<$count[0];$i++) //iterate based on column count
                {
                    if($i>76) break;

                    if($count[1] && $i==0) // Exception for image column, if there are images
                        $event->sheet->getColumnDimension('A')->setWidth(17);
                    else
                        $event->sheet->getColumnDimension($columnindex[$i])->setAutoSize(true);
                }

                //Set row height
                for($i=0;$i<$count[1];$i++) //iterate based on row count
                {
                    $event->sheet->getRowDimension($i+2)->setRowHeight(60);
                }

                if($count[1])
                foreach($drawings as $k=>$drawing_temp)
                {
                    if($drawing_temp)
                    {
                        if($img_file = StoreImageUrlTmp($drawing_temp.'-h100'))
                        {
                            $drawing = new Drawing();
                            $drawing->setName('image');
                            $drawing->setDescription('image');
                            $drawing->setPath($img_file);
                            $drawing->setHeight(70);
                            $drawing->setOffsetX(5);
                            $drawing->setOffsetY(5);
                            $drawing->setCoordinates('A'.($k+2));
                            $drawing->setWorksheet($event->sheet->getDelegate());
                        }
                    }
                }
            },
        ];
    }

    public function collection()
    {
        return collect($this->data);
    }
}

Example how I call the reporting class:
\Excel::store(
                new \App\Exports\ReportExcel(array_keys($items[0]),$items, $excel_item_imgs),
                $filename_woext.'.xlsx',
                'local_public_downloads',
                \Maatwebsite\Excel\Excel::XLSX);

I'm afraid in your case that's the way to go then. Glad you managed to work around it!


[Enhancement] Enable automatic migration for first-time users and optionally for returning users

Description
This PR introduces a db migration service that automatically applies Alembic migrations for first-time users. Returning users can also have migrations applied automatically, depending on the configured environment settings.
Related Issue
Closes AGE-506

Failing tests fix here.


Add claim calculation

Add claim calculation to:

School > claims views

check your answers
claim details

Support > organisation > claims  and Support > claims views

check your answers
claim details

 Add in the funding calculation to the Check your answers claim page


feat: set capella to epoch 0

Changelog picked up from commits here:
feat: set capella to epoch 0

Once https://github.com/kurtosis-tech/eth-network-package/pull/61 merged this can also be merged


MA - Missing ORC21 message

Describe the bug
There was one message that errored out in the file received on 3/30/2022.  The error was caused by missing Ordering Facility Name in ORC.21, and Result Rpt/Staus Chng – Date/Time in OBR.22. Please review, edit, and resubmit as necessary.
Message ID: dd88b7a2-bb29-4510-8654-fe518965f0b4
Sample:<PHONE_NUMBER>
Error: Expected Required field Organization Name (ORC.21.XON.1 of type ST) is not found.
Error: Ordering Facility Name (ORC.21 of type ORC.21.XON.T) value is invalid.
Error: Expected Required field Results Rpt/Status Chng - Date/Time + (OBR.22 of type OBR.22.TS_CMP) is not found.

The file associate with the message Id is no longer in Azure Storage
Using ./prime sender-files command got error:
./prime sender-files --env prod --message-id dd88b7a2-bb29-4510-8654-fe518965f0b4 -o ./junk/ticket5060
Exception Error:         Error using the report-files API
Status Code: 503
Message: Service Unavailable
Details:

CDC Test Page

Centers for Disease Control and Prevention
Test Page
This is a test page for prime.cdc.gov.
This page will appear if an error or timeout occurs at origin.
Reference ID:

-----
I also login into the Azure Storage and search for the file listing in database: https://pdhprodstorageaccount.blob.core.windows.net/reports/receive%2Fprescryptive.default%2Fprescryptive-covid-19-10a3e22a-cadc-40e6-b0f7-c4fc4932094d-20220329152224.hl7.  I can't find it either.  Therefore, I will close this ticket.


feat(helpers): extract messages from API error in case of 400

Part of work of #507
covers only helpers package
Checklist

[x] I followed code and contributing guidelines
[x] I wrote all needed tests

great work! 👍


Add missing providers.

Add missing providers.

:tada: This issue has been resolved in version 1.2.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Optimization required on ES service during ES High CPU usage  ( Data loss )

Hi Team ,
On last week ,One of our  ODES (open distro Elastic search) cluster  experienced high CPU load (NewRelic) and caused the indices to be out of sync (temporarily outdated, but eventually recovered).
We are suspecting  that ES itself is not optimized, and during high loads, this causes delay and missing data while syncing data from MongoDB.
Please can you suggest how to optimize the ODES during High CPU usage or How to optimize the Mongo connector Index

Hi there, it's true that high periods of sustained CPU load can cause Elasticsearch to throw errors, etc. Part of the solution is just throwing more hardware at the Elasticsearch cluster. The other part is having some sort of retry logic in your indexer (although in this case, it sounds like it recovered over time).


Use GET to access OSRM in place of POST

Use GET to access OSRM in place of POST, since there is nothing to POST and OSRM 4.8.1 return error on POST to /table

This was a workaround to avoid long queries being truncated because of length limitations with GET. Related posts for OSRM:
https://github.com/Project-OSRM/osrm-backend/issues/1293
https://github.com/Project-OSRM/osrm-backend/pull/1502
I will check out to the latest 4.8.1 and try to figure out the best way to deal with long queries.

Seriously thinking about using libosrm now...

But, using libosrm force using a local OSRM.

I think POST should be ok with table requests on OSRM 4.8.1. From what I get from 1502, the change that broke things for the current osrm_wrapper code is the injection of a ? into the request (clearly visible from osrm-routed output).

But, using libosrm force using a local OSRM

Yep, this is the only drawback.

Have you tried using a remote OSRM instance with the current code? I would appreciate some feedback on this as I'm always working on the same machine.

I'm only test with remote OSRM

I'm not sure if I understand all the pieces. But a POST request must contains a body. It's not the case with osrm_wrapper, always use the query.

Yes, implementing POST queries would really require to do things properly in the wrapper. I'll just let things as they are for now as it is not really a limitation for vroom use.

I'm only test with remote OSRM

Ok thanks, so it is working fine in this setting.


Module not found: can't resolve 'angular2-multiselect-dropdown/angular2-multiselect-dropdown' in '...'

I can find the module in the node-modules, and webstorm can goto it by goto definition, but it raise this error message while doing "npm start".
Dose anybody know how to resolve this problem?

try this： import { AngularMultiSelectModule } from 'angular2-multiselect-dropdown';

try this： import { AngularMultiSelectModule } from 'angular2-multiselect-dropdown';

it works, thanks very much.

Please update your README in that case. It says
import { AngularMultiSelectModule } from 'angular2-multiselect-dropdown/angular2-multiselect-dropdown';
there which is incorrect!

Guys, please update the README.
Thanks

Fixed ...
On Thu, Feb 28, 2019 at 6:14 AM Leonardo Ferreira de Moura <
<EMAIL_ADDRESS>wrote:

Guys, please update the README.
Thanks
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/CuppaLabs/angular2-multiselect-dropdown/issues/324#issuecomment-468089908,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADxjoAZIIm-wuaSMfxsVKvhQTCmmdYiaks5vRyZzgaJpZM4X5ZIu
.

--
Thanks & Regards
Pradeep Kumar Terli
Bangalore, India
https://cuppalabs.github.io

The import statement should be
import { AngularMultiSelectModule } from 'angular2-multiselect-dropdown';
On Wed, Feb 20, 2019 at 3:09 PM Murray Furtado<EMAIL_ADDRESS>wrote:

Please update your README in that case. It says
import { AngularMultiSelectModule } from
'angular2-multiselect-dropdown/angular2-multiselect-dropdown';
there which is incorrect!
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/CuppaLabs/angular2-multiselect-dropdown/issues/324#issuecomment-465499320,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADxjoIR8VrL24NKfFlGQo_Sve4Br5psoks5vPRfigaJpZM4X5ZIu
.

--
Thanks & Regards
Pradeep Kumar Terli
Bangalore, India
https://cuppalabs.github.io


Remove institute name in signup page

Is your feature request related to a problem? Please describe.
Remove institute name in the signup page as it is no longer needed
Component: SignUpComponent

I can work on this issue.


Commit dist folder

This merge request commits the dist folder to avoid the HACS action failing on pull request from forked repos due to the missing releases (example). For download, HACS keeps prioritising the releases over the dist folder.

coverage: 81.969% (+0.3%) from 81.714%
when pulling 3b4b500523ceaa431c3b9106eee47af3326fa6b9 on commit_dist_folder
into 998861017e46fa38b89596a99c3bb5e22a887a3a on master.


Supported hardSwish.

@huningxin PTAL, thanks.

I ran this code locally with same steps as CI's , it passed on "npm run build-production --if-present" step where CI failed, @huningxin  any suggestion, thanks.


How to send cloudevent?

What is the question:
Is there a cloudevent receiving address? I tried configuring SYNAPSE_CLOUDEVENTS_SINK_URI and it didn’t seem to work

@cdavernas Thanks for your answer, but it seems does not works for me.

workflow use examples:
{
  "id": "eventbasedgreeting",
  "name": "Event Based Greeting Workflow",
  "description": "Event Based Greeting",
  "version": "1.0.0",
  "specVersion": "0.8",
  "events": [
    {
      "name": "GreetingEvent",
      "source": "greetingEventSource",
      "type": "greetingEventType"
    }
  ],
  "functions": [
    {
      "name": "greetingFunction",
      "operation": "file://myapis/greetingapis.json#greeting",
      "type": "rest"
    }
  ],
  "start": "Greet",
  "states": [
    {
      "name": "Greet",
      "type": "event",
      "onEvents": [
        {
          "actions": [
            {
              "functionRef": {
                "refName": "greetingFunction",
                "arguments": {
                  "name": "${ .greet.name }"
                }
              }
            }
          ],
          "eventRefs": [
            "GreetingEvent"
          ],
          "eventDataFilter": {
            "data": "${ .greet }",
            "toStateData": "${ .greet }"
          }
        }
      ],
      "stateDataFilter": {
        "output": "${ .payload.greeting }"
      },
      "end": true
    }
  ]
}

when running,  but no trigger logs

@cdavernas  do you have any suggestions? thanks

By the way, the logs displayed in the UI only concern the workflow pod/process it is related to. In case of issues, you might one to check the server's, too.

@hongliang5316 Yes, indeed! You could submit a PR to fix it, but AFAIK its not the sole problem with those examples.

@cdavernas
Ok, I will do more tests later, and if I encounter problems, I will propose and make a PR.


[connectivity_plus_web] fix Bad state: Stream has already been listened to

…
Description
This was also changed on the original connectivity package but I think it happened right after this package was copied. See https://github.com/flutter/plugins/blob/connectivity-v3.0.6/packages/connectivity/connectivity_for_web/lib/src/dart_html_connectivity_plugin.dart
Checklist
Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]).
This will ensure a smooth and quick review process.

[x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs.
[x] My PR includes unit or integration tests for all changed/updated/fixed behaviors (See [Contributor Guide]).
[x] All existing and new tests are passing.
[x] I updated the version in pubspec.yaml and CHANGELOG.md.
[x] I updated/added relevant documentation (doc comments with ///).
[x] The analyzer (flutter analyze) does not report any problems on my PR.
[x] I read and followed the [Flutter Style Guide].
[x] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Does your PR require plugin users to manually update their apps to accommodate your change?

[ ] Yes, this is a breaking change (please indicate a breaking change in CHANGELOG.md and increment major revision).
[x] No, this is not a breaking change.

Thanks! I just requested a small change before merging.

Thank you! I will release this today


CMP implementation, incremental PR chunk 4: CMP context/parameters and utilities

Certificate Management Protocol (CMP, RFC 4210) extension to OpenSSL
Also includes CRMF (RFC 4211) and HTTP transfer (RFC 6712)
CMP and CRMF API is added to libcrypto, and the "cmp" app to the openssl CLI.
Adds extensive man pages and tests.  Integration into build scripts.
Incremental pull request based on OpenSSL commit 8869ad4a39f of 2019-04-02
4th chunk: CMP context/parameters and utilities
in crypto/cmp/cmp_ctx.c, crypto/cmp/cmp_util.c, and related files
Checklist

[x] documentation is added or updated
[x] tests are added or updated

BTW, I'm on vacation for the next two weeks (until June 23rd),
but Andreas likely will be able to react to review comments and perform requested changes.

rebased to openssl/master; fix conflicts in util/libcrypto.num

@Akretsch has rebased this PR again such that currently there are no conflicts any more.
The Travis CI issue currently shown is unrelated to our PR.

@FdaSilvaYY, thanks for your comments about 16 days ago.
Did you notice that @Akretsch reacted to them two weeks back?
Anything still to be done from your perspective?

@FdaSilvaYY, thanks for your comments about 16 days ago.
Did you notice that @Akretsch reacted to them two weeks back?
Anything still to be done from your perspective?

I'm Ok with his answers, nothing to add.
My remarks are not blocking ones, as I'm not part of project.

@mattcaswell, on May 8th you suggested for the preview of this chunk (https://github.com/mpeylo/cmpossl/pull/178#discussion_r282113262) to use the trace API for CMP logging (info/warning/error etc. output).
It turns out that this is not suitable because the trace API is statically disabled by default, so would not be usable for normal/production builds.
So I'm not sure whether it's worth continuing with the extracted enhancements of the trace API I proposed in PR#9110.
We're currently re-working this aspect of our CMP contribution such that warnings etc. are buffered in the CMP context and can later be output, using any means, at application level.

you suggested for the preview of this chunk (mpeylo#178 (comment)) to use the trace API for CMP logging (info/warning/error etc. output).
It turns out that this is not suitable because the trace API is statically disabled by default, so would not be usable for normal/production builds

Ok, if the trace API is not viable then we need a rethink.

So I'm not sure whether it's worth continuing with the extracted enhancements of the trace API I proposed in PR #9110.

Perhaps not.

We're currently re-working this aspect of our CMP contribution such that warnings etc. are buffered in the CMP context and can later be output, using any means, at application level.

Ok - that sounds interesting. I'd like to understand that proposal a little better.

Thanks @mattcaswell for your swift response.

So I'm not sure whether it's worth continuing with the extracted enhancements of the trace API I proposed in PR #9110.

Perhaps not.

So I tend to close that PR unless anyone objects there.
In case it turns out to be of interest later, I suppose it could be re-opened.
BTW, does the OpenSSL project have a backlog?

We're currently re-working this aspect of our CMP contribution such that warnings etc. are buffered in the CMP context and can later be output, using any means, at application level.

Ok - that sounds interesting. I'd like to understand that proposal a little better.

We plan for two things depending on the type of info/warning.

The PKIStatus info included in the server's reply is already stored in the CMP_CTX: ctx->lastPKIStatus.
In case this is not 'accepted' and there is no actual error then a respective warning message can be generated and output at application level, e.g., in apps/cmp.c.
We also have other sources of warnings and infos. For these we plan to simply provide string fields such as ctx->warnings and ctx->infos, which can be incrementally appended to line-by-line by the lib and then output at application level.

Of course it would be simpler to output warnings and infos right away (as we did so far, using a callback stored in the CMP_CTX), but the crypto lib seems not to support this, maybe because the idea is that the lib itself should be silent or because warnings are not considered of interest.
Another option could be to extend the existing error queue mechanism to support warnings (and possibly further levels of severity/verbosity.
What do you think?

Hmm, I just realized that buffering status info messages for output after return is not always adequate.
For instance, intermediate reports of the CMP session handler such as "sending ir" and "received response" should be output right away.
Otherwise, when a CMP transaction, which may involve multiple request/response pairs, gets stuck the user or calling application does not obtain any indication how far the transaction has been able to proceed.

Ok, for timely output (such as INFO: got response) that is more for experimenting and debugging we can make use of the existing trace API (though it unfortunately needs to be enabled statically (e.g., using ./config enable-trace).
Therefore I suppose the solutions sketched above should be sufficient and not hard to implement.
@mattcaswell, do you agree or do you have further comments/advice?
We'd like to continue&finish implementing this tomorrow.

@mattcaswell, do you agree or do you have further comments/advice?

It sounds reasonable to me.

Another option could be to extend the existing error queue mechanism to support warnings (and possibly further levels of severity/verbosity).

I'm unsure about that option. Probably it strays a little far from the original scope of what we are attempting to do here.

A technique used in several other parts of OpenSSL is to allow the application to register a callback that gets progress information.  For example while generating prime numbers...

A technique used in several other parts of OpenSSL is to allow the application to register a callback that gets progress information. For example while generating prime numbers...

Yeah, this is what we have been doing so far, but then Matt suggested using the trace API.
Which I tried, but then it turned out that it is disabled by default due to data sensitivity concerns.
So we could either go back to the original (callback-based) solution or do the mix of solutions sketched above.

For what my opinion is worth, I think using common practice rather than a mix of solutions will be easier for consumers to understand.

@mattcaswell, would it be ok for you if we revert to the original version,
which uses a pointer of the following type in CMP_CTX?
typedef int (*OSSL_cmp_log_cb_t) (const char *component,
                                  const char *file, int lineno,
                                  OSSL_CMP_severity level, const char *msg);

@mattcaswell, can we go back to using the CMP-specific logging callback?
Would be nice if you could provide your view within an hour or so since we need to know which way to proceed to today, and progress with this chunk 4 and the upcoming preview of chunk 5 is currently blocked due to the open choice/decision.
Like Rich, we'd prefer the callback variant for its flexibility and convenience of use.

can we go back to using the CMP-specific logging callback?

I'm not keen on a CMP specific logging callback. If logging is a requirement it seems like it should not be a CMP specific thing. That's just my view though so it would be interesting to hear opinions from others.
@openssl/committers, @openssl/omc.

Unfortunately nobody has responded so far.
Some more thoughts from my side:
A general configurable OpenSSL-wide logging facility would sound good, but it would be something to be introduced before chunk 4 can be finished.
And where to place configuration such as a callback 'globally'? In CMP we simply use the CMP_CTX.
Also I'm not sure how suitable it would be to have a shared configuration for such a logging facility.

Unfortunately nobody has responded so far.

Sorry, I‘ll try to post my thoughts sometimes this evening.

And if such a global logging callback is implemented, it must be able to replace all the existing callbacks (portably of course), else it's just adding +1 to an existing stack of things.

Sorry, I‘ll try to post my thoughts sometimes this evening.

Working on it. Please give me another day...

all the existing callbacks

@richsalz could you please give some examples which specific callbacks you have in mind?

SSL_CTX_set_info_callback, SSL_set_info_callback, BN_generate_prime, the new tracing stuff, BIO_set_callback, etc.
grep callback doc/man3/* shows lots of potential files.

SSL_CTX_set_info_callback, SSL_set_info_callback, BN_generate_prime, the new tracing stuff, BIO_set_callback, etc.

Ah, ok. You have the really big picture in mind.

I think any new API in this area should be able to handle all of the use-cases already in use, yes. Otherwise it's just making the API bigger and missing a chance to simplify things.

In case you wanna have a look at the CMP-specific logging facility we've been using in the preview of this PR https://github.com/mpeylo/cmpossl/pull/178 until (according to Matt's comment of May 8th: https://github.com/mpeylo/cmpossl/pull/178#discussion_r282113262) I tentatively switched this to the trace API on June 5th,
you can find it as the code portions marked in red (removal) in https://github.com/mpeylo/cmpossl/commit/52b9e81c2dbb5816ae783432babbc89cad6f0973
There we haven been using severity levels inspired by those in syslog.h and crypto/bio/bss_log.c:
typedef enum {OSSL_LOG_EMERG, OSSL_LOG_ALERT, OSSL_LOG_CRIT, OSSL_LOG_ERR,
              OSSL_LOG_WARNING, OSSL_LOG_NOTICE, OSSL_LOG_INFO, OSSL_LOG_DEBUG}
   OSSL_LOG_severity;

A general configurable OpenSSL-wide logging facility would sound good, but it would be something to be introduced before chunk 4 can be finished.
And where to place configuration such as a callback 'globally'? In CMP we simply use the CMP_CTX.
Also I'm not sure how suitable it would be to have a shared configuration for such a logging facility.

I'm not so sure whether adding a new logging API to the OpenSSL library is a good idea. Logging is a very application centric concept, but OpenSSL is not an application library. It's a low level library which provides algorithms and protocols intended to be used by applications, not to provide application logic by itself. We need to be careful not to overburden libcrypto with features which deviate from the main target of simply being a cryptographic library.
Currently, libcrypto already has an error reporting API and a tracing API. The error reporting might not be flexible enough at the moment, but IMHO it would be better to enhance it than to introduce yet another API.
Some Terminology
Before I can explain my viewpoint, I need to introduce some terminology. Because everybody uses and understands terms like 'tracing' and 'logging' in a different way. So we need a common vocabulary before being able to discuss the topic. The terminology represents just my personal point of view, it's not an official OpenSSL terminology.
There are many possible ways of communication between a C library and an application (resp. its user). Here I distinguish the following three types:

Error Reporting
Tracing
Logging

The first two types already exist in OpenSSL, the third one is what you are seeking.
Error Reporting
OpenSSL follows the traditional pattern of low level C APIs by indicating a function's failure via exceptional return values and leaving some additional diagnostic information on the 'error stack' for the application to pick up. It lies in the responsibility of the application to obtain and interpret the information, and to report it to he user.
Tracing
With tracing I mean a special form of diagnostic output (most often sent to stderr or a file, sometimes to an application window) which helps the developer understand and troubleshoot some otherwise hard to understand behaviour of the library. Typical properties of tracing are

It is only available in a special 'debug' build and not part of the release build.
The diagnostic output is very detailed and technical. Often it reports function entries and exits, parameter and variable values, and source code locations, information which is only intellegible if the source code is available.
The output can be very verbose and can have performance impacts (sometimes even if it's not enabled).

Logging
Logging is very similar to tracing, but it targets a different, less technical audience. Typical properties are

Logging is available in the release build. Most often it is partially or fully disabled by default and it requires some runtime configuration to turn it on or make it more verbose.
The output describes what the library is doing in a way such that a normal user or technically experienced engineer can understand what's going on. Messages are classified by severity in order to help the user distinguish between important and not so important information. A verbosity level can be used to tune how much of the not so important information he likes to see.
A library with good logging capabilities reports events and important information about those events, not function entries and exits.
If the logging addresses the regular user, internationalization of the messages might be an issue.

Logging and tracing have a lot in common, they could even rely on the same techniques (e.g. in OpenSSL: using BIOs as communication channels). The two most important differences are the targeted audience and the way to enable it (build time vs. runtime).
The Current State
While error reporting was part of the official API from the very beginning, tracing was introduced only recently. Not that there was no tracing code before that, but it was available only as a personal gadget for some programmers. You needed to edit the source code or #define some preprocessor constant on the compiler command line in order to activate it.
The new OpenSSL tracing API is an attempt to reactivate and unify the existing trace code which was lying idle. In some cases, one could even argue that the code was resurrected from oblivion and probably not used for years. It is my opinion that we still need to do a screening before the 3.0 release in order to identify the useful tracing categories and discard the others. The tracing API is a very new feature and still needs to mature. It was a deliberate decision to not include the tracing code in the release builds. Not only because of possible perfomance impacts, but also for security considerations. And it gives us more freedom to let the API evolve and mature.
Implementing an official logging API is a completely different calibre than just adding some tracing. It requires much more forethought and needs to be designed carefully in order to be flexible and extensible. Also, it should be useful for the entire library, not only to specific parts of it. In particular, it should be powerful enough to implement features like sophisticated logging and diagnostics of the TLS handshake, or diagnostic output from providers.
Currently, I don't see the capabilities to design and implement such a logging API in view of the large efforts for the new provider concept and the FIPS certification in version 3.0. So for the moment, I don't see an alternative to using the tracing API and accepting the fact that it can be used only by developers. It's better than nothing and it leaves you with the option to migrate from the tracing to the logging API if the latter will ever get implemented.
Sorry for my pessistic point of view. Maybe other team members think differently about it.

Thanks a lot @mspncp for your fundamental response!
Good that you clarified the terminology, to which I generally agree.
For CMP we need three aspects:

Error reporting, as usual. The only issue we face here is that the size of error messages has a limit of 4096 characters, which can be exceeded when reporting, e.g., details about a chain of certificates and the contents of the trust store in case a cert verification fails. Our current solution, as part of this PR for CMP chunk 4, is to split such over-long texts into multiple messages (see OSSL_CMP_add_error_txt()).
Alternatively, we might switch output of these details to the trace API, with the drawbacks that this is not available in release builds and that the details would be somewhat disconnected from the error message they belong to. Here in my view the drawbacks would be too severe, so I'd like to keep using the error queue with the workaround until the limit is relieved.
Progress reporting. In this case our need is not detailed diagnostics (like with typical tracing), but to report immediately, e.g., that a request message is about to be sent or that a response message has been received, where multiple such events can occur before the CMP session terminates. In case the session gets stuck and the send/receive info was buffered (as done for error reporting), there would be no output and it would remain unclear how far the session progressed before it got stuck or was killed. Here logging would be the best option. Since it is not (yet) available in OpenSSL, we'll have to resort to tracing, which unfortunately is not available in release builds. I think we can cope with this drawback for this use case at least for now.
Warnings and Info reporting. Also here logging would obviously be the best solution. Without it, we have to resort to some internal buffering of such information, for instance in the CMP context, which can then later be retrieved at application level and printed/forwarded suitably. I just had a thought that we could also abuse the error queue to store non-error messages (with a prefix such as "WARN" and "INFO") and request the application to output the error queue also in case there as no error. Would this be ok with you? In the longer run, the OpenSSL error queue could be extended with official tags such as 'warning' and 'info' to be stored with queue entries and to be printed in appropriate manner along with (actual) error messages.

In a brief survey of existing logging frameworks for C, @Akretsch found the following:

https://github.com/wonder-mice/zf_log
http://hardysimpson.github.io/zlog/
http://log4c.sourceforge.net/

Maybe these could be of help when - at some more suitable time in the future - designing and implementing a proper OpenSSL logging facility.
BTW, by when is version 3.0 currently expected?

BTW, by when is version 3.0 currently expected?

We hope to be code complete by the end of this year.

The error stack is just limited to 16 entries, so this will just push older
entries off the stack sooner or later.
My feeling regarding such large amounts of error text, would be that they
might be better handled as some kind of diagnostic structure, together with
conversion functions from struct to BIO or FILE.  But it would be good
to be able to use and react to that information in the application program
as well.

Well, printing the certs etc. to the error queue at least serves the most direct needs, namely that a person deploying the CMP client can find out in detail what does wrong and why. Such information has proved to be extremely helpful because there are many and partly pretty subtle reasons why a certificate validation does not succeed, an in particular inexperienced users then to make all kinds of mistakes here.
I don't expect an automatic analysis of such errors, for which a more structured output would be of course much more suitable.
So far, I have not been hurt by the limit of 16 error messages, but how about relieving this number to, e.g., 64?

BTW, such a detailed error diagnosis on cert path validation errors could be of great benefit also for (most) other cert verification use cases.
How about generally enhancing X509_verify_cert() that way?

We meanwhile have reverted the CMP logging to make use of the current vanilla trace API und did some further improvements to it.
@mattcaswell, could you please have a look again?
@bernd-edlinger or @openssl/committers, who is willing to do a second review of this CMP chunk?

can you have a look at the CI build failuers?

Unfortunately we had some code and doc inconsistencies w.r.t. the forth and back regarding the trace API.
These are fixed now and so this CMP chunk is ready again for further review :)

I just found that we still had a couple of inconsistencies w.r.t. reverting the tentative changes to the trace API, which are now eliminated.
Moreover we added a function to the CMP-internal utilities (for the benefit of later chunks).

@mattcaswell and @bernd-edlinger,
when do you think you'll be able to continue reviewing this chunk?

@mattcaswell et al.,
like we did for the previous CMP chunk in order to minimize the overall contribution period,
I've just provided a preview of chunk 5 such that it can be pre-reviewed already in parallel to this chunk 4: https://github.com/mpeylo/cmpossl/pull/178

Rebased and aligned  to #9058

Thanks @mattcaswell for all your yesterday's comments!
We've started handling them.

For one comment:  https://github.com/openssl/openssl/pull/9107#discussion_r305787317 for some reason I am not offered a reply button/textbox, so I'm quoting it here in order to respond:

Hmmm...I find the current state of affairs quite confusing. The conventions we typically follow are:

Use OSSL_ prefix for new types (where we're not using an established alternative prefix such as EVP_, SSL_ etc)
Use the name of the primary object at the start of the function name
Capitalise the object name where its a public function. Use lowercase otherwise.

With the above I find the use of CMP_CTX_* quite confusing - since there is no CMP_CTX object. It should really be OSSL_CMP_CTX_* for public functions or ossl_cmp_ctx_* for internal functions. If converting all existing uses to lowercase for internal functions is just too hard, then I'd personally prefer everything was OSSL_CMP_CTX_*.

So far we thought that a (new) function should have the  OSSL_ prefix if (and only if) it is public/exported, and for this reason we (mostly) dropped the OSSL_ prefix for internal functions.
Now we learn that there is an exceptional case:
If the function name is prefixed by the name of an exported type, this prefix should be in all-uppercase (which is already the case for our functions starting with OSSL_CMP_CTX_) if the function is exported, and this prefix should be in all-lowercase if the function is internal - do we now understand this correctly?
This means that we have to re-add ossl_ to the internal functions starting with the type name prefix CMP_CTX_ and make this type name prefix lowercase, such that they end up having the prefix ossl_cmp_ctx_.
For us it does not make a difference whether we rename into upper case or lower case; the main effort is to consistently do the rename action also in any related (also internal) documentation and to make sure that the layout rules are not broken when the length of a function name changes.

The exceptional case is when the prefix is used in an object oriented fashion like a class name, followed by a method name. In this case, the public methods are nanme as CLASS_NAME_method_name(), whereas the  private methods are named class_name_method_name().  (See the RAND_DRBG "class" for an example.)

So far we thought that a (new) function should have the OSSL_ prefix if (and only if) it is public/exported, and for this reason we (mostly) dropped the OSSL_ prefix for internal functions.

No - there are many examples of ossl_ prefix being used internally, e.g. we have the type OSSL_PROVIDER. External functions that operate on that type have the form OSSL_PROVIDER_something, whereas internal functions have the form ossl_provider_something.

If the function name is prefixed by the name of an exported type, this prefix should be in all-uppercase (which is already the case for our functions starting with OSSL_CMP_CTX_) if the function is exported, and this prefix should be in all-lowercase if the function is internal - do we now understand this correctly?

Yes - that is correct.

This means that we have to re-add ossl_ to the internal functions starting with the type name prefix CMP_CTX_ and make this type name prefix lowercase, such that they end up having the prefix ossl_cmp_ctx_.

Sorry for all the churn in this area!! We really need to update our coding style to write all these rules down.

Rebased  to openssl/master

@mattcaswell et al., finally we've handled all your comments.
Further comments (or even better: approval) would be appreciated.

@DDvO do you have any clue what went wrong with the CI builds?

@DDvO do you have any clue what went wrong with the CI builds?

Thanks for pointing out this.
It was last last night when I did the last push.
It turns out there is a bug in initialization of the trace facility causing an infinite recursion. I've reported this and provided a fix in #9567.
Moreover, there is a limitation of the C preprocessor of clang, which I've tried to work around now. Let's see how it copes with the new version...

I noticed the mention "Submodule pyca-cryptography deleted from 094031 ",
but I cannot comment it in place ;)
local changes unwanted commit ?

Oops, yes, this was an inadvertent commit (I had tried out techniques to ignore the submodules).
Thanks for pointing this out; fixed.

Now the PR is clean again :)

Thanks @mattcaswell and @FdaSilvaYY for your recent comments!
I've just finished addressing all of them.
Anything more to do on this PR?

Thanks @bernd-edlinger for your thoughtful comments!
I've addressed all of them, fixing all those partly pretty subtle pointer management issues.

@bernd-edlinger, this was very productive again :)
I've handled all your new comments, but I cannot hide/close them (and neither other finished comments) because this PR was submitted by @Akretsch.

I'll be on vacation until Sep 9,
while @Akretsch will return on Aug 29 and can then handle any further comments.

rebased to openssl/master

@openssl/committers @mattcaswell We would like to have a second reviewer for your PR #9107 .

Added "pending 2nd review" label (sorry should have done this before). Ping @openssl/committers.

Please update the copyright/license statements to all follow the required style. I've provided an example adapted to the noted context. The form should be Copyright  Company/Project for consistency.
There is one place where the copyright text is repeated and at least one place using the older pre-Apache License wording. They should be made consistent.

rebased to openssl/master

@bernd-edlinger You reviewed already some CRMF/CMP related code at a functional level. Again we need a 2nd reviewer. Maybe you could have a look to this PR?

Sure, no problem.
But could you please have a look at my comments recent comments an #7646
https://github.com/openssl/openssl/pull/7646#discussion_r317433078 https://github.com/openssl/openssl/pull/7646#discussion_r317433518
and https://github.com/openssl/openssl/pull/7646#discussion_r31743326 https://github.com/openssl/openssl/pull/7646#discussion_r317433315 ?
Thanks!

rebased to openssl/master

rebased to openssl/master

@bernd-edlinger  Hopefully all of your comments in #7646 are fixed now. I asked @mattcaswell for a 2nd review of #9774.

@bernd-edlinger, after some further minor cleanup this PR is ready again for any further comments (or hopefully 2nd approval).

Rebased to openssl/master + small fixes

Pushed to master! Thanks all!

65-test_cmp_ctx is causing some test failures on some local test machines when enable-trace option is specified.
1..1
# Subtest: /home/nse/workspace/openssl-linux_x64_GCC/build-linux_x64_gcc/test/cmp_ctx_test
1..46
ok 1 - test_CTX_reinit
ok 2 - test_CTX_set_get_option_16
ok 3 - test_CTX_set_get_log_cb
# ERROR: (int) 'test_log_cb_res == 1' failed @ ../test/cmp_ctx_test.c:276
# [0] compared to [1]
not ok 4 - test_cmp_ctx_log_cb
E0:E6:24:BE:7C:7F:00:00:error:CMP routines:(unknown function):multiple san sources:../test/cmp_ctx_test.c:139:
ok 5 - test_CTX_print_errors
===============
/usr/local/bin/perl ../Configure --prefix=/home/nse/workspace/openssl-linux_x64_GCC/install shared enable-trace linux-x86_64
Perl information:
/usr/local/bin/perl
5.24.0 for x86_64-linux

Enabled features:
aria
asm
async
autoalginit
autoerrinit
autoload-config
bf
blake2
camellia
capieng
cast
chacha
cmac
cmp
cms
comp
ct
deprecated
des
dgram
dh
dsa
dso
dtls
dynamic-engine
ec
ec2m
ecdh
ecdsa
engine
err
filenames
fips
gost
idea
legacy
makedepend
md4
mdc2
module
multiblock
nextprotoneg
pinshared
ocb
ocsp
padlockeng
pic
poly1305
posix-io
psk
rc2
rc4
rdrand
rfc3779
rmd160
scrypt
seed
shared
siphash
siv
sm2
sm3
sm4
sock
srp
srtp
sse2
ssl
static-engine
stdio
tests
threads
tls
trace
ts
ui-console
whirlpool
tls1
tls1-method
tls1_1
tls1_1-method
tls1_2
tls1_2-method
tls1_3
dtls1
dtls1-method
dtls1_2
dtls1_2-method

Disabled features:
ktls                    [default]        OPENSSL_NO_KTLS
afalgeng                [too-old-kernel]
asan                    [default]        OPENSSL_NO_ASAN
buildtest-c++           [default]
crypto-mdebug           [default]        OPENSSL_NO_CRYPTO_MDEBUG
crypto-mdebug-backtrace [default]        OPENSSL_NO_CRYPTO_MDEBUG_BACKTRACE
devcryptoeng            [default]        OPENSSL_NO_DEVCRYPTOENG
ec_nistp_64_gcc_128     [default]        OPENSSL_NO_EC_NISTP_64_GCC_128
egd                     [default]        OPENSSL_NO_EGD
external-tests          [default]        OPENSSL_NO_EXTERNAL_TESTS
fuzz-libfuzzer          [default]        OPENSSL_NO_FUZZ_LIBFUZZER
fuzz-afl                [default]        OPENSSL_NO_FUZZ_AFL
md2                     [default]        OPENSSL_NO_MD2 (skip crypto/md2)
msan                    [default]        OPENSSL_NO_MSAN
rc5                     [default]        OPENSSL_NO_RC5 (skip crypto/rc5)
sctp                    [default]        OPENSSL_NO_SCTP
ssl-trace               [default]        OPENSSL_NO_SSL_TRACE
ubsan                   [default]        OPENSSL_NO_UBSAN
unit-test               [default]        OPENSSL_NO_UNIT_TEST
uplink                  [no uplink_arch] OPENSSL_NO_UPLINK
weak-ssl-ciphers        [default]        OPENSSL_NO_WEAK_SSL_CIPHERS
zlib                    [default]
zlib-dynamic            [default]
ssl3                    [default]        OPENSSL_NO_SSL3
ssl3-method             [default]        OPENSSL_NO_SSL3_METHOD

Config target attributes:
AR => "ar",
ARFLAGS => "r",
CC => "gcc",
CFLAGS => "-Wall -O3",
CXX => "g++",
CXXFLAGS => "-Wall -O3",
HASHBANGPERL => "/usr/bin/env perl",
RANLIB => "ranlib",
RC => "windres",
asm_arch => "x86_64",
bn_ops => "SIXTY_FOUR_BIT_LONG",
build_file => "Makefile",
build_scheme => [ "unified", "unix" ],
cflags => "-pthread -m64",
cppflags => "",
cxxflags => "-std=c++11 -pthread -m64",
defines => [  ],
disable => [  ],
dso_ldflags => "-z defs",
dso_scheme => "dlfcn",
enable => [ "afalgeng" ],
ex_libs => "-ldl -pthread",
includes => [  ],
lflags => "",
lib_cflags => "",
lib_cppflags => "-DOPENSSL_USE_NODELETE -DL_ENDIAN",
lib_defines => [  ],
module_cflags => "-fPIC",
module_cxxflags => "",
module_ldflags => "-Wl,-znodelete -shared -Wl,-Bsymbolic",
multilib => "64",
perl_platform => "Unix",
perlasm_scheme => "elf",
shared_cflag => "-fPIC",
shared_defflag => "-Wl,--version-script=",
shared_defines => [  ],
shared_ldflag => "-Wl,-znodelete -shared -Wl,-Bsymbolic",
shared_rcflag => "",
shared_sonameflag => "-Wl,-soname=",
shared_target => "linux-shared",
thread_defines => [  ],
thread_scheme => "pthreads",
unistd => "<unistd.h>",

Recorded environment:
AR =
ARFLAGS =
AS =
ASFLAGS =
BUILDFILE =
CC =
CFLAGS =
CPP =
CPPDEFINES =
CPPFLAGS =
CPPINCLUDES =
CROSS_COMPILE =
CXX =
CXXFLAGS =
HASHBANGPERL =
LD =
LDFLAGS =
LDLIBS =
MT =
MTFLAGS =
OPENSSL_LOCAL_CONFIG_DIR =
PERL =
RANLIB =
RC =
RCFLAGS =
RM =
WINDRES =
__CNF_CFLAGS =
__CNF_CPPDEFINES =
__CNF_CPPFLAGS =
__CNF_CPPINCLUDES =
__CNF_CXXFLAGS =
__CNF_LDFLAGS =
__CNF_LDLIBS =

Makevars:
AR              = ar
ARFLAGS         = r
CC              = gcc
CFLAGS          = -Wall -O3
CPPDEFINES      =
CPPFLAGS        =
CPPINCLUDES     =
CXX             = g++
CXXFLAGS        = -Wall -O3
HASHBANGPERL    = /usr/bin/env perl
LDFLAGS         =
LDLIBS          =
PERL            = /usr/local/bin/perl
RANLIB          = ranlib
RC              = windres
RCFLAGS         =

@slontis, I wonder why your build issue pops up here, after this PR has already been merged.
I cannot reproduce it with the current OpenSSL master and (essentially) the configuration you mention:
./Configure shared enable-trace linux-x86_64

No problems.. I  will debug it on one of the machines it fails on (hopefully without using printf :) )
Just asking in case something stood out as being a obvious issue.
I also could not get it to happen on my local machine.


feat(generator/console): add make:controller command

This PR is a starting point of creating the make commands for the framework using the internal Generation Component.
Here's the related issue #472
It add a make:controller command

it take a classname as an argument like Test
it will suffix the class with Controller like TestController.php
it will prepend the main autoloaded namespace/path from the composer file to it
it will then glue them with a Controllers directory
it will generate the right namespace PSR-4 compatible
it support adding subfolders directly in the classname arg from the command

So, for example if I have a project with a default autoloaded namespace from composer like :
my-project/app => App
And I run make:controller Admin/Dashboard
It will generate a class named DashboardController at my-project/app/Controllers/Admin/DashboardController.php
And with the following namespace App\Controllers\Admin

Next steps before duplicate this into others make commands are :

[ ] Replace the way to retrieve the root path of the final user project
[ ] Find a way to test the file is properly generated and at the right location, with the right classname and namespace
[ ] Extract the main generation logic to the Generator component
[ ] ( later ) refactor the file creation logic by the Filesystem component
[ ] Handle Dummy replacements in Stubs file
[ ] Add MakeControllerCommand specifics params like the route path and the view path

For the 2 first TODOs I'm looking for ideas, if someone have any.

@innocenzi For now, I'm not sure of the purpose of the ClassGenerator but I think the main logic could go there right ?

@brendt are we good moving this outside the framework to the CLI project?

I haven't tested this locally, but as I understand it, it will generate a controller in a predefined directory, right?
I don't really like this, personally, and I don't think it follows the philosophy of Tempest, where everything can be anywhere.
My implementation of tempest publish prompts the user every time for the path at which the file would be published. If the file already exists, the user is asked if the file should be replaced. What do you think of this approach?

No problem for me, I can follow your already approved work, I'll check how your command work !
Then, we assume that every make command will always ask for the dist folder, we could imagine to give the main namespace path as an indication in the prompt if the user has no idea.
I'll try to go in this way ( must think about how to generate the right namespace for the file then, maybe show a select for all composer.json autoload or ? )

are we good moving this outside the framework to the CLI project?

Will discuss on GitHub

Pull Request Test Coverage Report for Build<PHONE_NUMBER>4
Details

53 of 206 (25.73%)  changed or added relevant lines in 8 files are covered.
10 unchanged lines in 5 files lost coverage.
Overall coverage decreased (-1.4%) to 80.589%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/Tempest/Console/src/Stubs/ControllerStub.php
0
2
0.0%

src/Tempest/Support/src/PathHelper.php
29
39
74.36%

src/Tempest/Console/src/Commands/Generators/MakeControllerCommand.php
0
56
0.0%

src/Tempest/Console/src/Commands/PublishCommand.php
0
85
0.0%

Files with Coverage Reduction
New Missed Lines
%

src/Tempest/Generation/src/SimplifiesClassNames.php
1
90.77%

src/Tempest/Auth/src/CreatePermissionsTable.php
2
75.0%

src/Tempest/Auth/src/CreateUserPermissionTable.php
2
77.78%

src/Tempest/Auth/src/CreateUsersTable.php
2
81.82%

src/Tempest/Generation/src/ManipulatesPhpClasses.php
3
69.33%

Totals

Change from base Build<PHONE_NUMBER>4:
-1.4%

Covered Lines:
6813

Relevant Lines:
8454

💛  - Coveralls

Don't worry about commits, I've pulled the unmerged work of Enzo to refactor the code with its helpers on namespaces and ClassManipulator.
Then I tried to stick as much as possible to the implementation of the publish command.
So actually there is lot of duplicate code between both which is good, because it means we can now abstract those mechanism to create commands that generate files faster.
I'll look into that in the followings days.

As discussed in Discord here https://discord.com/channels/1236153076688359495/1295433932027990027/1298306566335893525
This PR is "ready", we are waiting to see the organization of the "roadmap" for the feature

Let's close this one in favor of #647


get() should return a Message instance or NULL

Right now get() returns a MethodBasicGetOkFrame instance (or MethodBasicGetEmptyFrame instance after #14). From the developers perspective that is way too lowlevel. When I call get(), I expect to get a Message instance (or NULL when there is no message in the queue). Just like when I register a consumer, I expect the callback to get a Message instance and not a Frame instance.
Moreover the Frame returned by get() does not contain the payload which is probably caused by this.
Being very new to Bunny I have no idea why the payload is not being set. And even if it was, I still don't know how to process it to get the message contents. I can provide a PR but I need some advice :)

get() isn't implemented property at the moment, I'll look into that. Thank you for reporting this. However for real use I suggest consume() with message handler as it's more performant.

The get() is now implemented properly and returns Message in both sync and async clients. Hope this helps.


chore: added missing imports

Summary

Added missing imports to address the forum post... maybe.

Checklist

[ ] I added — tests to prove my changes
[ ] I updated — all the necessary docs
[X] I reviewed — the entire PR myself, using the GitHub UI
[ ] I described — all breaking changes and the Migration Guide

I'd be surprised if this resolved that post but the change is still valid 👍🏻

Also doubtful - a guy can dream though!


Takedown request from Netcraft

I've been informed by DigitalOcean's security team that it has received a notification from Netcraft of an ongoing malware attack. It's sending me a takedown notice. The email is publicly shared below.
I have responded, saying that there is no malicious software on Bore and linking them to this repository. If anyone has advice on how we can continue to keep bore running or can reach out to people at DigitalOcean or Netcraft, please let me know. It would be a shame if this meant that we couldn't keep offering bore as a public service to people; it's been very useful.

Hi,
We are writing to let you know that your Droplet bore.pub  at <IP_ADDRESS> is hosting active malicious software (malware) or performing activity similar to known malware. We haven’t taken any action on the droplet for now.
You can review the full notification we received here:  https://digitalocean.abusehq.net/share/oYfJwNGDOXVGwrxssXzrcnc2Y6UzUfK5YLVzf95lnhiAycQh3NhgDMnaFsOno7en
Please investigate and remediate the situation if you have access to the Droplet to do so, or relay this information to your client so they may do so. You may share the URL or its contents with them if it will help expedite their investigation.
There are numerous tools and options available for you to investigate, but we have found that ClamAV ( http://www.clamav.net/ ) is a good tool to initially attempt to identify and remove the malware and is recommended for scanning your Droplet. Most linux distributions have ClamAV in their package management system, so you should only need to install it and then run it.
Please treat this matter urgently and investigate as soon as possible. You will need to respond to us with the steps that you have taken to remove the malware in question, and let us know how you have secured the Droplet to prevent repeated occurrences. We review all tickets after 24 hours. If we do not hear from you and the activity is ongoing, we may have to block port 80 and 443 on your droplet to prevent further harm until the issue is resolved.
If you have any questions or concerns regarding this, please reply to this ticket and we’ll be happy to assist in any way we can.
Regards,
Security Operations
DigitalOcean Security
You can view the abuse history for this Droplet bore.pub at  https://digitalocean.abusehq.net/share/y-GoDos65eY7nlWnF4Il_Q.
For a full overview of abuse reports regarding your account please visit https://digitalocean.abusehq.net/share/2pUIrpf6cb3MorcW06miKwxoEj7ukwAx_w2bGG330YM.
ref:!00Df2018t5m.!500QP0QngVN:ref

I've received the following response.
To whoever is potentially using bore for distributing malware, please don't do this!

Hello,
We appreciate your attention to this issue. We will consider this matter resolved, and merely ask that you continue to respond to and address any reports of abuse we may forward you for this Droplet.
If you have any questions, please do not hesitate to let us know.
Regards,
Security Operations Center
DigitalOcean

Confirmed that it's working.
If you're getting an HTTP 502 error, that is not on our side — bore doesn't know how to speak HTTP.


Add Publish, Target Local Installs

Added asynchronous credentials handling as per https://github.com/Nordstrom/artillery-plugin-aws-sigv4/pull/9#issuecomment-338091930.


Detected

Unauthorized extension use detected

[Violation] 'requestAnimationFrame' handler took ms
[Violation] 'requestAnimationFrame' handler took 61ms
[Violation] 'requestAnimationFrame' handler took 92ms
[Violation] 'requestAnimationFrame' handler took 60ms
[Violation] 'requestAnimationFrame' handler took 58ms
[Violation] 'requestAnimationFrame' handler took 70ms
[Violation] 'requestAnimationFrame' handler took 101ms
[Violation] 'requestAnimationFrame' handler took 83ms
[Violation] 'requestAnimationFrame' handler took 52ms
[Violation] 'requestAnimationFrame' handler took 109ms
[Violation] 'requestAnimationFrame' handler took 61ms
[Violation] 'requestAnimationFrame' handler took 112ms
[Violation] 'requestAnimationFrame' handler took 138ms
[Violation] 'requestAnimationFrame' handler took 149ms
[Violation] 'requestAnimationFrame' handler took 181ms
[Violation] 'requestAnimationFrame' handler took 153ms
[Violation] 'requestAnimationFrame' handler took 59ms

Is looped also may be the cause of freezing.
_I would recommend remaking the entire code on another language because this is deprecated

Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user's experience. For more help, check https://xhr.spec.whatwg.org/.


[Bug]: Embeddings Deletion Causes "Delete of nonexisting embedding ID"

What happened?
Hi there, I tried to upload two PDF files to a persistant collection and delete one of them. But I received Warning Messages: "Delete of nonexisting embedding ID". This Warning only appears when I upload multiple files and delete one of them. Here are my test files and code.
alphabet-2023-q1-10q.pdf
Apple Inc.-10K.pdf
from langchain.text_splitter import CharacterTextSplitter
from langchain.document_loaders import PyPDFLoader
import chromadb
from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction

def main():
    # create collection
    client = chromadb.PersistentClient(path="./chroma_db")
    collection = client.get_or_create_collection(name="test", embedding_function=OpenAIEmbeddingFunction())
    text_splitter = CharacterTextSplitter(chunk_size=1500, chunk_overlap=100)

    # load document_1
    loader_1 = PyPDFLoader("alphabet-2023-q1-10q.pdf")
    documents1 = loader_1.load()
    docs_1 = text_splitter.split_documents(documents1)
    ids_1 = [str(i) for i in range(1, len(docs_1) + 1)]
    texts_1 = [split.page_content for split in docs_1]
    metadatas_1 = [split.metadata for split in docs_1]
    collection.add(ids=ids_1, metadatas=metadatas_1, documents=texts_1)

    # load document_2
    loader_2 = PyPDFLoader("Apple Inc.-10K.pdf")
    documents_2 = loader_2.load()
    docs_2 = text_splitter.split_documents(documents_2)
    ids_2 = [str(i) for i in range(47, len(docs_2) + 47)]
    texts_2 = [split.page_content for split in docs_2]
    metadatas_2 = [split.metadata for split in docs_2]
    collection.add(ids=ids_2, metadatas=metadatas_2, documents=texts_2)

    print(f"ids_1: {ids_1}")
    print(f"ids_2: {ids_2}")

    print("count before", collection.count())
    # delete document_1
    collection.delete(ids_1)
    print("count after", collection.count())

if __name__ == '__main__':
    main()

Versions
chromadb==0.4.5
langchain==0.0.264
python==3.10.12
MacOS==13.3.1
Relevant log output
ids_1: ['1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31', '32', '33', '34', '35', '36', '37', '38', '39', '40', '41', '42', '43', '44', '45', '46']
ids_2: ['47', '48', '49', '50', '51', '52', '53', '54', '55', '56', '57', '58', '59', '60', '61', '62', '63', '64', '65', '66', '67', '68', '69', '70', '71', '72', '73', '74', '75', '76', '77', '78', '79', '80', '81', '82', '83', '84', '85', '86', '87', '88', '89', '90', '91', '92', '93', '94', '95', '96', '97', '98', '99', '100', '101', '102', '103', '104', '105', '106', '107']
count before 107
Delete of nonexisting embedding ID: 1
Delete of nonexisting embedding ID: 2
Delete of nonexisting embedding ID: 3
Delete of nonexisting embedding ID: 4
Delete of nonexisting embedding ID: 5
Delete of nonexisting embedding ID: 6
Delete of nonexisting embedding ID: 7
Delete of nonexisting embedding ID: 8
Delete of nonexisting embedding ID: 9
Delete of nonexisting embedding ID: 10
Delete of nonexisting embedding ID: 11
Delete of nonexisting embedding ID: 12
Delete of nonexisting embedding ID: 13
Delete of nonexisting embedding ID: 14
Delete of nonexisting embedding ID: 15
Delete of nonexisting embedding ID: 16
Delete of nonexisting embedding ID: 17
Delete of nonexisting embedding ID: 18
Delete of nonexisting embedding ID: 19
Delete of nonexisting embedding ID: 20
Delete of nonexisting embedding ID: 21
Delete of nonexisting embedding ID: 22
Delete of nonexisting embedding ID: 23
Delete of nonexisting embedding ID: 24
Delete of nonexisting embedding ID: 25
Delete of nonexisting embedding ID: 26
Delete of nonexisting embedding ID: 27
Delete of nonexisting embedding ID: 28
Delete of nonexisting embedding ID: 29
Delete of nonexisting embedding ID: 30
Delete of nonexisting embedding ID: 31
Delete of nonexisting embedding ID: 32
Delete of nonexisting embedding ID: 33
Delete of nonexisting embedding ID: 34
Delete of nonexisting embedding ID: 35
Delete of nonexisting embedding ID: 36
Delete of nonexisting embedding ID: 37
Delete of nonexisting embedding ID: 38
Delete of nonexisting embedding ID: 39
Delete of nonexisting embedding ID: 40
Delete of nonexisting embedding ID: 41
Delete of nonexisting embedding ID: 42
Delete of nonexisting embedding ID: 43
Delete of nonexisting embedding ID: 44
Delete of nonexisting embedding ID: 45
Delete of nonexisting embedding ID: 46
count after 61

Process finished with exit code 0

@HammadB

I have the problem too

@tazarov Hi, could you please look at this problem? Thank you for you time!

@mickey-lyx, thanks for reporting this. I'll take a look at this soon. At a glance, the code looks fine, and the actual result seems to be fine - you have 61 docs once you remove 47 from the starting 107. All in all, this seems like a warning, not an actual bug. The I will have a look and let you know.

@tazarov Really appreciate it. The result is right. I'm just wondering why there appears to be warnings of deleting nonexisting embeddings. Is it because the embeddings were deleted multiple times?

@tazarov
I have the problem too.
but, in chromadb 0.3.29 is no problem.
fixed it by downgrading the version.

I have the same issue, and running queries on the db triggers this warning every time. What I did is selected items based on where statement (no ID was given) and removed them one-by-one:
my_collection.delete(
            where={"file_id": str(file_id)}
        )

Since then the warning is shown every time I query it.

I'm having the same issue. This seems to occur even when an empty list is passed as ids to Collection.delete.

We'd love to get this fixed - is anyone able to help post a minimal repro?

@jeffchuber
import chromadb
from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction

def main():
    client = chromadb.PersistentClient(path="./chroma_db")
    collection = client.get_or_create_collection(name="test", embedding_function=OpenAIEmbeddingFunction())

    num_1 = 47
    num_2 = 70

    texts_1 = [f"text_1.{i}" for i in range(num_1)]
    ids_1 = [f"1.{i}" for i in range(num_1)]
    texts_2 = [f"text_2.{i}" for i in range(num_2)]
    ids_2 = [f"2.{i}" for i in range(num_2)]

    collection.add(ids=ids_1, documents=texts_1)
    collection.add(ids=ids_2, documents=texts_2)

    print("count before", collection.count())
    collection.delete(ids_1)
    print("count after", collection.count())

if __name__ == '__main__':
    main()

I'm seeing similar warnings, but I'm unsure if I should be concerned since it's a warning. It would be good to get some insights to why this occurs even after uploading a few PDF files.
112-49d5-a776-2c02c03897e8:77661df1-86bc-4f33-9119-a90d77f7c24e
chroma               | 2023-09-16 15:22:31 WARNING  chromadb.segment.impl.vector.brute_force_index Delete of nonexisting embedding ID: c441314d-7112-49d5-a776-2c02c03897e8:484a228b-de38-4674-8f14-078f4f218afd
chroma               | 2023-09-16 15:22:31 WARNING  chromadb.segment.impl.vector.brute_force_index Delete of nonexisting embedding ID: c441314d-7112-49d5-a776-2c02c03897e8:51c75801-6ecd-4490-941e-8ee6f2229476
chroma               | 2023-09-16 15:22:31 WARNING  chromadb.segment.impl.vector.brute_force_index Delete of nonexisting embedding ID: c441314d-7112-49d5-a776-2c02c03897e8:282cb350-257b-49ef-ae55-ab3997099d58
chroma               | 2023-09-16 15:22:31 WARNING  chromadb.segment.impl.vector.brute_force_index Delete of nonexisting embedding ID: c441314d-7112-49d5-a776-2c02c03897e8:fe9d8119-b72a-44c1-9bc5-f5c173621a4b
chroma               | 2023-09-16 15:22:31 WARNING  chromadb.segment.impl.vector.brute_force_index Delete of nonexisting embedding ID: c441314d-7112-49d5-a776-2c02c03897e8:c92f759d-f0e7-46e9-9156-e5c47e917de7
chroma               | 2023-09-16 15:22:31 WARNING  chromadb.segment.impl.vector.brute_force_index Delete of nonexisting embedding ID: c441314d-7112-49d5-a776-2c02c03897e8:5be4bf1c-7c02-4815-9c25-de4463b0231f
chroma               | 2023-09-16 15:22:31 WARNING  chromadb.segment.impl.vector.brute_force_index Delete of nonexisting embedding ID: c441314d-7112-49d5-a776-2c02c03897e8:32500766-ceb7-4b12-8e8d-04b34306f30f
chroma               | 2023-09-16 15:22:31 WARNING  chromadb.segment.impl.vector.brute_force_index Delete of nonexisting embedding ID: c441314d-7112-49d5-a776-2c02c03897e8:7e5d60fd-cb8a-4ecf-adf3-8d86694458e8
chroma               | 2023-09-16 15:22:31 WARNING  chromadb.segment.impl.vector.brute_force_index Delete of nonexisting embedding ID: c441314d-7112-49d5-a776-2c02c03897e8:5cfbdc44-cc08-4749-8d5d-d628f6aa4676
chroma               | 2023-09-16 15:22:31 WARNING  chromadb.segment.impl.vector.brute_force_index Delete of nonexisting embedding ID: c441314d-7112-49d5-

package versions
chromadb==0.4.10
langchain==0.0.225
Running chroma client server with the latest Docker version
  chroma:
    container_name: chroma
    image: ghcr.io/chroma-core/chroma:latest
    volumes:
      - index_data:/chroma/chroma
    environment:
      - IS_PERSISTENT=true
      - CHROMA_SERVER_HTTP_PORT=8000
    restart: unless-stopped
    ports:
      - '8000:8000'
    networks:
      - projectgpt

I have the same issue, and running queries on the db triggers this warning every time. What I did is selected items based on where statement (no ID was given) and removed them one-by-one:
my_collection.delete(
            where={"file_id": str(file_id)}
        )

Since then the warning is shown every time I query it.

I am having this exact issue too

@HammadB I think we can close this now.

I think this issue is still present. I've just stumbled upon it in my application. And I'm using latest (0.4.24) version of Chroma, so the fix from #1150 should probably be already merged.

我更新了chromadb==0.5.0,但还是有这个问题：
我是用threading更新的：
t=threading.Thread(target=mydb.add_collection_from_file,args=[local_f],daemon=True)
t.start()

@running-frog, @s-peryt, we have a bug in the HNSW binary index that, under certain conditions, can result in the above errors. There is a PR  - #2062 that should resolve this.


In 711 handle multiple configs

Starting DXBuilder conversion
Start adapting run_workflows
Move functions around to dx_utils

This change is

Hello @Yu-jinKim! Thanks for opening this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file resources/home/dnanexus/run_workflows/utils/dx_requests.py:

Line 899:80: E501 line too long (80 > 79 characters)
Line 913:80: E501 line too long (104 > 79 characters)
Line 935:80: E501 line too long (84 > 79 characters)
Line 981:80: E501 line too long (84 > 79 characters)
Line 989:80: E501 line too long (96 > 79 characters)

In the file resources/home/dnanexus/run_workflows/utils/dx_utils.py:

Line 202:9: E128 continuation line under-indented for visual indent

In the file resources/home/dnanexus/run_workflows/utils/utils.py:

Line 312:80: E501 line too long (82 > 79 characters)


Require approved client apps or app protection policy with mobile devices

Hello,
About this one: Require approved client apps or app protection policy with mobile devices
Shouldn't it be targeted only to Exchange and not to All Apps?

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 14348ba7-48f0-1862-ebec-0fc3640e587d
Version Independent ID: 8188cee4-3f28-6014-f3be-1ee9e9b183b6
Content: Conditional Access - Require approved app or app protection policy - Azure Active Directory - Microsoft Entra
Content Source: articles/active-directory/conditional-access/howto-policy-approved-app-or-app-protection.md
Service: active-directory
Sub-service: conditional-access
GitHub Login: @MicrosoftGuyJFlo
Microsoft Alias: joflore

@lightupdifire
Thanks for your feedback! We will investigate and update as appropriate.`

@MicrosoftGuyJFlo
Well, I have read about 10 Tonn of MS articles, and there are some 1% MS articles pointing to the situation, as an example:

If you would like to open a yellow door
Push the green button
Not explaining a 100 of combinations what can be done around in order to open a yellow door.
When looking for a specific solution, for a specific policy, it would be very nice to have a specific clear explanation... If this is article is form another 99% articles, then yeah go ahead, close that case amigo :)

@lightupdifire I will add a little clarity in a future update. Thanks for the feedback.


[FLINK-36368] Do not prematurely merge CommittableManager [1.19] (#25405)

Unchanged backport of #25659.

Reviewed by Chi on 05/12/24. In hand.


latest pyscf support scf calculation under background charge?

dear developers,
The latest pyscf support scf calculation under background charge? thank you.

@leebo-2020, yes it does, please see the example below:
https://github.com/pyscf/pyscf/blob/85addafa1697ad128d401922e123c66f85215daa/examples/qmmm/00-hf.py#L10-L33
Just to be sure I checked with my locally installed version (<IP_ADDRESS>) and the example runs without a problem. Searching for QMMM in the GitHub actions section also shows that the tests are working with the latest version of the master branch.


fix: :bug: sync using origin remote by default

I believe this change should fix the errors related to tokens being expired or overall changes to the config not being used after the first clone.
Instead of relying explicitly or implicitly in the origin remote, I used the URL that we already combine every time as a parameter to the <repository> argument of the fetch and pull commands.
I tested this with a simple testcase that I created on purpose, if someone could please test this further (preferably with a real issue - the ones regarding the error code 128), that would be awesome.

@jbonadiman can you list down the tests that you did as well?

This should address #53


fix(schema) fix validation of nested records in metaschema

Includes a regression test in the metaschema testsuite, and a testcase in the
schema testsuite (which demonstrates that the nested records issue was a
metaschema-only problem).
Includes some minor corrections to the metaschema specification, detected
using the metaschema to attempt to validate itself... unfortunately the schema
validator doesn't implement a type system strong enough to fully validate the
metaschema without falling into an infinite loop -- in fact, the schema library isn't strong enough to validate schemas without having the metaschema
"cheat" and perform some of its validation of schemas using the
check escape-hatch function to recurse into records, as we see in this
commit. This is by design, because the schema library is meant to be only as complex as necessary for handling Kong entities; the additional complexity is tucked away in the metaschema, which remains a non-essential nice-to-have. Still, the metaschema is useful enough to catch syntax errors in schema definitions, so it's good to keep it around.
As a bonus, there's an extra commit that simplifies some code of the schema module.

Maybe refactor instead of optimize?

@thibaultcha I went with optimize cause I'm pretty sure it moves less bits around :)

Then perf is our scope of choice :) (Actually that's my bad, I meant to write perf but my comment mentions refactor for some reason).