[RFE] Add storage contol plane related scenarios to ocp wrapper

Is your feature request related to a problem? Please describe.
It would be good to have some scenarios that create PVCs, bind them to PVs, mount them in pods and finally reclaim them.
Describe the solution you'd like
PV/PVC related scenarios on OCP wrapper with appropriate checks that a storageclass exists in case dynamic provisioning is reqired.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

Related to https://github.com/cloud-bulldozer/kube-burner/issues/426

Can you define what you mean by scenario?

@mfleader "scenario" just a generic term for a new workload. Like one alongside those here: https://github.com/cloud-bulldozer/kube-burner/tree/master/cmd/kube-burner/ocp-config

Do the pods exist before the scenario starts?
From my initial work, it seems like you cannot mount a pvc volume by patching an existing pod.

No. Think of it as you will create everything for the workload. Attaching a volume to an existing pod isn't a valid use case. Usually, you create PV/PVC first, then pod next and the pod mounts the volume.
Since we're interested in dynamic provisioning here, then we only need to create the PVCs and not the PVs. Maybe you'll have to create a StorageClass that supports dynamic provisioning. You'll have to do that analysis.

Our current workloads create everything and take measurements to measure the scaleup, then delete as an unmeasured "garbage collection", but we are working towards measuring GC as well.
If you simply follow that pattern, then we can compare the added effect/overhead of PVC mounting on top of what we already understand reasonably well (e.g. node-density with added PVC).
Usually creating a single pod doesn't consume enough resources for us to measure, but we could achieve that by using churn. It could be a second phase to figure out whether churn is beneficial here and what settings to use.

Should this live in cmd/kube-burner/ocp-config or in a different directory?

Yes: "PV/PVC related scenarios on OCP wrapper ..."

How do you do dynamic storage provisioning for a KinD cluster?
The obvious provisioners seem to be these OpenShift plugins for these platforms.

How do the other workloads 'scale'?

How are the labels supposed to be specified?

How do you verify the count of objects that have been created?

With regards to potential dynamic storage provisioners, the obvious ones seem to be these OpenShift plugins for these platforms.
Are there other platforms that we want to test that would need a dynamic storage provisioner not already available as one of the above plugins?

These seem to be a good enough and handful of provisioners as a part of our workload. And also I like this idea of parameterizing the STORAGE_PROVISIONER with a flexibility for user to specify it based on their requirements and infra.

How do the other workloads 'scale'?

https://github.com/cloud-bulldozer/kube-burner/blob/master/pkg/burner/create.go#L200

How are the labels supposed to be specified?

https://github.com/cloud-bulldozer/kube-burner/blob/master/pkg/burner/create.go#L207

How do you verify the count of objects that have been created?

https://github.com/cloud-bulldozer/kube-burner/blob/master/pkg/burner/utils.go#L82

Regarding mechanics and working of other workloads, I would suggest to explore the codebase and play around with it on a cluster to understand the common patterns and functionality across workloads/jobs. This is not something one could entirely help others with, requires some own research.
Docs: https://cloud-bulldozer.github.io/kube-burner/v1.7.8/

I feel like we might care about the PVC claim size to cluster/node total, or free, storage size ratio, but I'm unsure how to implement that currently.

Yes, its good to set a smaller value as default and expose claim size as configurable option to the end user. I see current value is being set to 1GB which is too high.
cc: @afcollins


Eaton Logo with Tagline

Describe the desired behavior
We need to add Eaton logos to the PX Blue icon set. These should include the plain Eaton logo and the one with the "Powering Business Worldwide" tagline.
Describe the current behavior
Basic icon is in the hidden section right now. The one with tagline does not exist.
Is this request related to a current issue?
This is related to requests coming from product teams about where to find the logos.

quite strange to put the tagline version in our repo... makes sense for the basic icon

Why does the small one make sense but not the big one? The big one is the official Eaton logo. The one without the tag line is only supposed to be used when the tag line one is too small to read. Both of these are by popular demand from adopters.

Just a feeling that it is less "icon" -- does the tagline one also has to be 24*24 view box?

If we want it to work well with the automatic creator utility for MUI Icons, yes. It might be a little goofy, but the EATON one will already be a little goofy since it's a rectangle too.


segfault when mport install on some packages

Reported via IRC:
mport install telegram-desktop
mport install lumina-desktop
both of these are causing a segfault.  The former does not exist in mports currently.   Not clear if it's the - in the name or what.
mport install foo correctly reports error

Fixed in the latest 2.2.9 release


Set root path to $HOME

The default root path when starting the livebook macOS app was /.
I think it would be better, if it is started with $HOME as root path.
The best solution would probably be a configurable root path from within the app.
@wojtekmach Is something like that possible with wxWidgets?

Thank you!


PROFESSIONAL REGISTRATION ISSUER CODE (MULTIDISCIPLINARY TEAM LEAD)

PROFESSIONAL REGISTRATION ISSUER CODE (MULTIDISCIPLINARY TEAM LEAD): https://www.datadictionary.nhs.uk/data_elements/professional_registration_issuer_code__multidisciplinary_team_lead_.html
has a broken link to Multidisciplinary Team Meeting.
It incorrectly goes to: https://www.datadictionary.nhs.uk/data_dictionary/nhs_business_definitions/m/multidisciplinary_team_meeting_de.asp?shownav=0
Fixed in the June 2023 release patch.
Check in the June 2023 release.

Issue now fixed.
Ticket closed.


Wrong LEAD-OUT in .cue file with multisession cd-rom / PX-760A drive

Version
Latest, 20220301
Describe the bug
With my drive/pc it seems all multisession cd-roms get an invalid .cue file when dumped with DIC. DIC says it's succesfully dumped and indeed all goes well, but the .cue / .bin files generated have file table problems when opened with IsoBuster or PowerIso. When dumped with ImgBurn, although it's not readed in raw mode, the cue/bin are functional.
It seems to be related with LEAD-OUT
This is the .cue file generated by DIC:
REM SESSION 01
FILE "TOVENARY (Track 1).bin" BINARY
  TRACK 01 AUDIO
    INDEX 01 00:00:00
REM LEAD-OUT 01:30:00
REM SESSION 02
REM LEAD-IN 01:00:00
REM PREGAP 00:02:00
FILE "TOVENARY (Track 2).bin" BINARY
  TRACK 02 MODE2/2352
    INDEX 01 00:00:00

This is the .cue file generated by ImgBurn for the same cd-rom:
FILE "tovenarij.bin" BINARY
REM SESSION 01
    TRACK 01 AUDIO
      INDEX 01 00:00:00
  REM LEAD-OUT 03:24:40
REM SESSION 02
    TRACK 02 MODE2/2352
      INDEX 01 05:56:40

This is the DIC .cue editted to make it open succesfully again:
REM SESSION 01
FILE "TOVENARY (Track 1).bin" BINARY
  TRACK 01 AUDIO
    INDEX 01 00:00:00
  REM LEAD-OUT 03:24:40
REM SESSION 02
REM LEAD-IN 01:00:00
REM PREGAP 00:02:00
FILE "TOVENARY (Track 2).bin" BINARY
  TRACK 02 MODE2/2352
    INDEX 01 00:00:00

So when using the LEAD-OUT value from ImgBurn it all works again. I'm not very technical at this but I hope this report is useful to help improving DIC
Screenshots
This is how it looks when opening with the original cue file from DIC (file table problems):

This is after fixing the LEAD-OUT manually to match imgburn's cuefile:

Disc title
Tovenarij! (Hybrid disc) cd-rom
URL
Archive.org
Log file
Log:
logs.zip

Use _img.cue, please. This cue is a general format, while .cue is a redump.org specific format. Other app can't support it completely.

So redump.org uses a specific .cue format that most popular iso software can not open? I was not aware of that.
Can't DIC perhaps have an option to create 2 .cue formats: one that works with generic iso software and one that fits the redump cue format? It seems in this case changing a few values makes it work again with popular iso software and that greatly helps people/archivists with little technical knowledge. But I'm not sure if it works with every multisession cd-rom. In my area about 1 of every 50 old secondhand cd-rom's are multisession, they may be rare but I still come across them quite often.
If it's a bug from IsoBuster, the dev is friendly and probably open to discuss it and perhaps improve the software
For anyone who is interested in this topic, someone pointed me to this discussion about the redump multisession cue format:

So DIC uses a specific .cue format that most popular iso software can not open?

Yes, multiple bin itself is the redump.org specific, though. I say again, please use the clone cd format (ccd+img+sub+cue) if you want to use the multi-session disc by the other app.

If it's a bug from IsoBuster

It's not a bug, IsoBuster merely has not supported the redump cue.

For anyone who is interested in this topic

PREGAP 02:32:00 is completely incorrect. Lead-out and Lead-in area, which it's 01:30:00 + 01:00:00, are not the pregap.

Sorry for late reply. Thanks for the answer, I'm closing the issue.

@saramibreak I am still not having any luck with mounting a DIC-produced multisession disc image with IsoBuster.
For example, I dumped this Counting Crows - Hard Candy Enhanced CD (multisession disc), and when I try to open the ccd/img/sub, it doesn't show the ISO9660 file system when it should exist:

If I try to open the _img.cue file, it doesn't work at all:

I also upgraded to IsoBuster 5.0, and have the same issues. In fact, now it does not show anything for the data track when opening the ccd file:

Counting Crows - Hard Candy_logs.zip

Use _img.cue, please. This cue is a general format, while .cue is a redump.org specific format. Other app can't support it completely.
I'm not an expert on this topic, but I think the multi-bin cues might not contain all the info necessary as the IsoBuster dev tried to add support for our cues. Would be great if you could get in dialog with him, he's a nice guy https://www.isobuster.com/support.php

Hi All,
I addressed this CUE 'issue' in IsoBuster 5.0 to the best of my knowledge.
Without having read the details in this thread, issues exist(ed) because the LEAD-OUT entry in the CUE is/was wrong or absent !
A correct LEAD-OUT is a must in all IsoBuster versions < 5.0  (also ImgBurn etc.) Without it and without it being correct, the second session address will be identified wrongly.  Do know that the address is also relative to the file start, so it depends on whether you have one or more files referenced by the CUE
A missing LEAD-OUT is now 'allowed' in IsoBuster >= 5.0 IF the second or later session is in its own file (or set of files).
IsoBuster is then usually able to get it right based on file length etc.
When I looked into this, triggered by people active in redump.org, and from what I remember (*), it was only an issue in CUE files.  I was not aware there were .CCD issues.  In fact I'd have to look into CCD altogether to refresh my memory () on the format.
(*) I have the memory of a goldfish
In case of the counting crows CD image.  Do you have 14 files ?  If so, try removing the REM LEAD-OUT entry from the CUE.
Does it work now in IsoBuster 5.0 ? (It won't in IsoBuster 4.9.1)

I might also add that I (in IsoBuster) was the very first person ever to support multi-session CUE+ISO/IMG/BIN files and as such I set the standard ..  So .. when I say it's wrong .. it's wrong !  LOL

In case of the counting crows CD image. Do you have 14 files ? If so, try removing the REM LEAD-OUT entry from the CUE.
Does it work now in IsoBuster 5.0 ? (It won't in IsoBuster 4.9.1)

I think that worked!

I wonder what would need to change for the img/ccd/sub to be parsed correctly, or for the cuesheet with the merged bin (img) to be parsed correctly.

Can you elaborate ?
With IsoBuster ?
With merged bin do you mean to say you'd like all files in one big file with a CUE ?
That is easy, since IsoBuster creates single file iso/bin files from a CD.
Right mouse click the top most CD/DVD icon in the left pane and select Extract
Make sure a CUE is created as well (settings) or create it separately (also via top icon right click)
IsoBuster will put the correct LEAD-OUT in the CUE (if IsoBuster correctly parses the file system - which it seems to do based on the screenshot)

Currently, IsoBuster is unable to parse the file system correctly based on the img/ccd/sub that is produced by DIC from a multisession disc. In other words, what would need to be changed in the ccd file?

Currently, IsoBuster is unable to parse the file system correctly based on the img/ccd/sub that is produced by DIC from a multisession disc. In other words, what would need to be changed in the ccd file?

Care to upload a set of files somewhere ?

Care to upload a set of files somewhere ?

https://1fichier.com/?rsbyaoyqddb1ngo6opnl

Thank you, I will have a look as soon as I can.
Question, why the two CUE variants ?  A CUE for a single *.img container and a CUE for a set of *.bin files ? (both flawed it seems)

@dutchcow can you reopen this issue?
@saramibreak thoughts?

Thank you, I will have a look as soon as I can.
Question, why the two CUE variants ? A CUE for a single *.img container and a CUE for a set of *.bin files ? (both flawed it seems)

https://github.com/saramibreak/DiscImageCreator/issues/121#issuecomment-1077882557

I'm going to have a look tomorrow, to try and figure things out.
For my understanding, is the *.ccd in its current form, as generated by DIC, actually properly supported by CloneCD ?
(If I'm not mistaken CCD is a CloneCD format)

OK, I've done my analysis based on the counting crows' CD images and here are the results.
I'm trying to be as objective as possible without pointing fingers.  This stuff has always been the wild west.
I'll break it up in three parts, to try and keep some level of overview.

CUE + multiple BIN files
CUE + single IMG file
CCD + single IMG file

The short:

CUE + multiple BIN files:
The DIC generated CUE's LEAD-OUT address is wrong

CUE + single IMG file:
The DIC generated CUE's LEAD-OUT address is correct but the DIC generated IMG is missing 11400 blocks

CCD + single IMG file:
The DIC generated CCD's last track address is wrong
On top of that there is an IsoBuster 5.0 64bit bug (not in 4.x nor 5.0 32bit) but it doesn't matter here since the CCD is wrong anyway

The Long:

CUE + multiple BIN files:

REM SESSION 01
FILE "Counting Crows - Hard Candy (Track 01).bin" BINARY
  TRACK 01 AUDIO
    ISRC USIR10211155
    INDEX 01 00:00:00
FILE "Counting Crows - Hard Candy (Track 02).bin" BINARY
  TRACK 02 AUDIO
    ISRC USIR10211068
    INDEX 01 00:00:00
FILE "Counting Crows - Hard Candy (Track 03).bin" BINARY
  TRACK 03 AUDIO
    ISRC USIR10211153
    INDEX 01 00:00:00
FILE "Counting Crows - Hard Candy (Track 04).bin" BINARY
  TRACK 04 AUDIO
    ISRC USIR10211157
    INDEX 01 00:00:00
FILE "Counting Crows - Hard Candy (Track 05).bin" BINARY
  TRACK 05 AUDIO
    ISRC USIR10211154
    INDEX 01 00:00:00
FILE "Counting Crows - Hard Candy (Track 06).bin" BINARY
  TRACK 06 AUDIO
    ISRC USIR10211151
    INDEX 00 00:00:00
    INDEX 01 00:00:65
FILE "Counting Crows - Hard Candy (Track 07).bin" BINARY
  TRACK 07 AUDIO
    ISRC USIR10211158
    INDEX 01 00:00:00
FILE "Counting Crows - Hard Candy (Track 08).bin" BINARY
  TRACK 08 AUDIO
    ISRC USIR10211159
    INDEX 01 00:00:00
FILE "Counting Crows - Hard Candy (Track 09).bin" BINARY
  TRACK 09 AUDIO
    ISRC USIR10211152
    INDEX 00 00:00:00
    INDEX 01 00:01:25
FILE "Counting Crows - Hard Candy (Track 10).bin" BINARY
  TRACK 10 AUDIO
    ISRC USIR10211150
    INDEX 00 00:00:00
    INDEX 01 00:01:70
FILE "Counting Crows - Hard Candy (Track 11).bin" BINARY
  TRACK 11 AUDIO
    ISRC USIR10211161
    INDEX 00 00:00:00
    INDEX 01 00:00:63
FILE "Counting Crows - Hard Candy (Track 12).bin" BINARY
  TRACK 12 AUDIO
    ISRC USIR10211160
    INDEX 00 00:00:00
    INDEX 01 00:00:35
FILE "Counting Crows - Hard Candy (Track 13).bin" BINARY
  TRACK 13 AUDIO
    ISRC USIR10211277
    INDEX 01 00:00:00
REM LEAD-OUT 01:30:00
REM SESSION 02
REM LEAD-IN 01:00:00
REM PREGAP 00:02:00
FILE "Counting Crows - Hard Candy (Track 14).bin" BINARY
  TRACK 14 MODE1/2352
    INDEX 01 00:00:00

First some necessary information:
1.1
A multi-session CUE needs a correct 'REM LEAD-OUT' entry to be properly supported.  I introduced multi-session support in CUE files probably ~20 years ago now and I laid down the law :)
IsoBuster versions <= 4.9 will not deal with the image correctly if the 'REM LEAD-OUT' entry is missing
Same for other applications, for instance ImgBurn:

1.1.1
One exception to 1.1: as of IsoBuster 5.0 the total absense of 'REM LEAD-OUT' is allowed in CUE + multi-file images.
The start address of the LEAD-OUT is then derived from the file-size associated with the last track of the session.
This works if the LEAD OUT truly follows the track.  If not, make sure to include a POSTGAP
1.2
MSF addresses are relative to the start of the file they're in.
I find that a bit counter-intuitive as well, especially when trying to preserve an identical layout, but it's how CDRWin's Jeff concieved it back in the days.
This is also why in this CUE file you will see INDEX 01 00:00:00 for every track, because every track is located in its own file.
The MSF address 00:00:00 is relative to the start of each file.
1.3
The ISO9660 file system and its derivatives (Joliet, Rock Ridge etc.) use absolute addressing internally !
In other words, an ISO9660 file system in a second session uses addressing relative to the start address of the first session (which is 0), not relative to the start address of the session it is in.
This is different for HFS for instance.  If the session containing an HFS file system gets assigned a different address from the original CD, it will still work, because it's relative to the start of the session.
Not for ISO9660 and Joliet.  This is why the address of the first data track in the second session here needs to be at the exact same address as the original.  If it's not, the ISO9660 file system will not parse correctly.
While this can be seen as a pest, in this case it's a blessing, because you have an excellent indirect way to test if the image is identical (in layout) than the original.  After all it is what led to this thread.
1.4
According to the various CD standards the pre-gap of the very first track on a CD starts at MSF: 00:00:00
MSF stands for Minutes:Seconds:Frames.  A second contains 75 frames.  A minute contains 60 seconds.  You do the math :)
The actual first Logical Block Address (LBA) of the track starts 150 blocks/frames later.  So LBA 0 = MSF 00:02:00
HOWEVER the creator of the CUE format, CDRWin's Jeff, in all his wisdom (ahum), decided to make LBA 0 = MSF 00:00:00 as well. Essentially ignoring the first track's pregap
This is just information that is not that important as long as you compare apples to apples !  As long as you're working with the CUE alone you don't need to worry about this too much.
However if you consider CDD as well and CloneCD Oliver's correct implementation of MSF, then you need to take this difference in account (see 3.)
As a consequence of 1.2 the 'REM LEAD-OUT' address is also relative to the start of the file associated with the last track of the session ( FILE "Counting Crows - Hard Candy (Track 13).bin" BINARY )
The 'REM LEAD-OUT' entry follows the 'FILE' entry and hence is part of it.
And since the disc LEAD-OUT immediately follows the track (here) the (relative to file) LEAD-OUT address is essentially the size of track 13.
After removing the 'REM LEAD-OUT' entry from this CUE (see 1.1.1), after loading the CUE in IsoBuster 5.0 and after being able to properly parse the ISO9660 file system (see 1.3) I now know the exact address of the first track of the second session.
I know the layout is properly identified now.  Based on this information I can also calculate the proper LEAD-OUT MSF address !  Namely the size of track 13.
Either by doing the math (see before) or by right mouse clicking the track in IsoBuster and by selecting "Extract From-To" where you can see the MSF size of the track:

BEFORE THE FIX:

FIX the LEAD-OUT address in the CUE, change it to:
REM LEAD-OUT 08:45:20
'Et Voila', the CUE + BIN files now works in IsoBuster 4.x and other applications (for instance ImgBurn)

CUE + single IMG file:

Counting Crows - Hard Candy_img.cue
FILE "Counting Crows - Hard Candy.img" BINARY
  REM SESSION 01
  TRACK 01 AUDIO
    ISRC USIR10211155
    INDEX 01 00:00:00
  TRACK 02 AUDIO
    ISRC USIR10211068
    INDEX 01 04:20:55
  TRACK 03 AUDIO
    ISRC USIR10211153
    INDEX 01 08:53:05
  TRACK 04 AUDIO
    ISRC USIR10211157
    INDEX 01 13:17:22
  TRACK 05 AUDIO
    ISRC USIR10211154
    INDEX 01 17:10:17
  TRACK 06 AUDIO
    ISRC USIR10211151
    INDEX 00 21:26:65
    INDEX 01 21:27:55
  TRACK 07 AUDIO
    ISRC USIR10211158
    INDEX 01 24:15:70
  TRACK 08 AUDIO
    ISRC USIR10211159
    INDEX 01 29:17:02
  TRACK 09 AUDIO
    ISRC USIR10211152
    INDEX 00 33:07:12
    INDEX 01 33:08:37
  TRACK 10 AUDIO
    ISRC USIR10211150
    INDEX 00 37:11:15
    INDEX 01 37:13:10
  TRACK 11 AUDIO
    ISRC USIR10211161
    INDEX 00 41:05:42
    INDEX 01 41:06:30
  TRACK 12 AUDIO
    ISRC USIR10211160
    INDEX 00 45:44:17
    INDEX 01 45:44:52
  TRACK 13 AUDIO
    ISRC USIR10211277
    INDEX 01 50:52:25
  REM LEAD-OUT 59:37:45
  REM SESSION 02
  TRACK 14 MODE1/2352
    INDEX 01 62:09:45

The CUE file appears to be correct.  The 'REM LEAD-OUT' entry is correct. The data track 'INDEX' address is correct too.  Both are relative to the start of the only file (Counting Crows - Hard Candy.img).  See 1.2 for the explanation.
We know it is correct because we have a working CUE + BIN files now (See 1.) (by either removing 'REM LEAD-OUT' or by fixing 'REM LEAD-OUT' and giving it address '08:45:20'
When we load that CUE in IsoBuster and right mouse click the top most CD icon in the left pane and select "Create Cue sheet file" we can generate a CUE for a single file image, because that is the format IsoBuster always chooses.
In that generated CUE file we can see the LEAD-OUT address matches the LEAD-OUT address in the DIC created CUE (same for data track INDEX).
So why doesn't it work ?  Because the IMG file misses 11400 blocks !
The session overhead, lead-out, lead-in and pregap of the first track in the second session are completely omitted from the image.
There is no mechanism in CUE + IMG image files to omit these blocks from an image file.  CUE knows PREGAP and POSTGAP, but they only apply to tracks and need to be placed at strict positions in the CUE
If you work with a single container file you cannot simply omit blocks from the middle, it messes with the addressing.  Any application will look for the second-session data at a wrong offset inside the file.
It would only be theoretically possible if a command would exist for it (say 'GAP 02:32:00).  But the standard does not provide this command and if an application chooses to use this mechanism it should at the very least introduce this (or a similar) command in the CUE to alert other applications.
As an FYI, the Clone CD *.CCD associated container image file ommits these 11400 blocks.  But CCD is another format in its own right.  It's not called CUE and it contains different data.  So an application is aware of this feature and can take it in account.
I realize this IMG is created for the associated CCD file in the file set !  And for the purpose of CCD it is created correctly.  But one should remove the CUE for it (in case of multi-session), since CUE and CCD expect different things from their container files (especially in the case of multi-session discs).  It just ads to the confusion.

CCD + single IMG file:

The IMG file is correct and created in a way that is expected for CCD files (see 2. last paragraph)
However the CCD file is not correct !
The start address of the data track in the second session is 870 blocks off from reality.  870 blocks is an odd number that I can't immediately correlate with something, so I'm not sure where it comes from ?
PS. CCD is just another text format (an INI file in fact)
The last '[Entry]' is the entry for the last track.  Which is the data track in the second session here.  In this image it's [Entry21]
[Entry 21]
Session=2
Point=0x0e
ADR=0x01
Control=0x04
TrackNo=0
AMin=0
ASec=0
AFrame=0
ALBA=-150
Zero=0
PMin=62
PSec=0
PFrame=0
PLBA=278850

IsoBuster uses the PMin, PSec and PFrame values to determine the track start address.
The PLBA value is ignored but needs to be a correct translation of the MSF address of course.
If you have a look at the IsoBuster Created CUE (See 2. second paragraph) you can see the correct address for the track.  It is: 279720 (REM LBA: 279720) or CUE-MFS 62:09:45 (INDEX 01 62:09:45)
See how I refer to the CUE-MSF here, because it's 150 frames or 2 seconds off from a true CD-standard MSF address (see 1.4)  In other words the true MSF address of the track is 62:11:45
Fix the CCD in this file set and you have a working CCD + IMG !!!  ((*) With one exception, see below)
[Entry 21]
Session=2
Point=0x0e
ADR=0x01
Control=0x04
TrackNo=0
AMin=0
ASec=0
AFrame=0
ALBA=-150
Zero=0
PMin=62
PSec=11
PFrame=45
PLBA=279720

(*) It appears IsoBuster 5.0 64-bit has an issue as it doesn't parse the CCD correctly !
The problem does not exist in the 32-bit version, nor in the lower versions (e.g. 4.x)
It's a conversion bug (5.0 is the first 64 bit version - it took quite a bit of work)
It was a simple fix and I have a test version for those who'd like it.  Otherwise you'll find it in next official Beta and of course IsoBuster 5.1

Multi-BIN: I think your understanding of REM LEAD-OUT differs to ours. You assume that REM LEAD-OUT is a total gap size between sessions. We assume that gap between sessions is a sum of 3 sizes:
REM LEAD-OUT
REM LEAD-IN
REM PREGAP
We can't change meaning of LEAD-OUT tag because it doesn't represent physical layout of the disc, even if it loads in IsoBuster.

CUE + multiple BIN files:
The DIC generated CUE's LEAD-OUT address is wrong

It's not wrong. (https://github.com/saramibreak/DiscImageCreator/issues/121#issuecomment-1077882557) (https://github.com/saramibreak/DiscImageCreator/issues/121#issuecomment-1078584116)
CUE + single IMG file:
The DIC generated CUE's LEAD-OUT address is correct but the DIC generated IMG is missing 11400 blocks

It's a specification. Other apps also do not generate 11400 blocks.
CCD + single IMG file:
The DIC generated CCD's last track address is wrong
On top of that there is an IsoBuster 5.0 64bit bug (not in 4.x nor 5.0 32bit) but it doesn't matter here since the CCD is wrong anyway

Surely the PSec and PFrame of your ccd file are wrong. But I dumped a multi-session disc by the latest release version and it's no problem. If you say it's a bug, use "Bug report" from "New Iisue", please.

Multi-BIN: I think your understanding of REM LEAD-OUT differs to ours. You assume that REM LEAD-OUT is a total gap size between sessions. We assume that gap between sessions is a sum of 3 sizes: REM LEAD-OUT REM LEAD-IN REM PREGAP

With specifications there really is only one interpretation.  The LEAD-OUT is an address.  It is where the lead-out starts, followed by a lead-in and pregap.  It is not a size or range, it is an address.  The address is relative to the start of the FILE entry that precedes it.

We can't change meaning of LEAD-OUT tag because it doesn't represent physical layout of the disc, even if it loads in IsoBuster.

I'm not sure what you mean by that.
Ok, so in the image @tjanas uploaded, the multi-BIN CUE says this:
REM LEAD-OUT 01:30:00
Please explain how you interpret this value ?
The way it needs to be interpreted, and the way other implementations interpret this (e.g. ImgBurn), the lead-out starts 1 minute and 30 seconds into FILE "Counting Crows - Hard Candy (Track 13).bin"
However, that file (track) is 8 minutes, 45 seconds and 20 frames long.
So .. how does that compute ?
If an application were to use 01:30:00 it would cut track 13 short, start the lead-out too soon and position track 14 wrong, which is essentially what happens and why other applications can't mount your CUE + multi-BIN
Mind you, in the single-IMG CUE the LEAD-OUT value is correct !

It's not wrong. (#121 (comment)) (#121 (comment))

Yes, multiple bin itself is the redump.org specific,

I see.
Then it's an unfortunate choice of file extension (*.cue).
Or at the very least REM LEAD-OUT could have been named slightly differently, so that apps understanding this command would not interpret it as intended and as a result misinterpret the format.

It's a specification. Other apps also do not generate 11400 blocks.

Apps do what they want with their own format, but should adhere to standards set for other formats.
You can't omit sectors from a container and expect the CUE to still work with it.
No application that interprets the CUE properly works with this format.
I tried a few.
Fact is that CCD works with omitting the 11400 sectors and so if you generate a correct CCD then that is fine.  But remove the *.CUE, because the CUE cannot work with this container (IMG) (when it's a multi-session disc)

Surely the PSec and PFrame of your ccd file are wrong. But I dumped a multi-session disc by the latest release version and it's no problem. If you say it's a bug, use "Bug report" from "New Iisue", please.

I worked with the uploaded image from @tjanas.  Download it and notice the address of the data track is recorded wrongly in the CCD file.  I can't comment on other releases or other CDs.  I'm just providing feedback on the files shown to me, to try and help out with feedback because the confusion on your community is huge.

I believe I used a fairly recent version of DIC to dump this disc. If there is a newer version than the one I used that will produce different cue/ccd files, I will give that a try and report back with those differences.

Multi-BIN: I think your understanding of REM LEAD-OUT differs to ours. You assume that REM LEAD-OUT is a total gap size between sessions. We assume that gap between sessions is a sum of 3 sizes: REM LEAD-OUT REM LEAD-IN REM PREGAP

With specifications there really is only one interpretation. The LEAD-OUT is an address. It is where the lead-out starts, followed by a lead-in and pregap. It is not a size or range, it is an address. The address is relative to the start of the FILE entry that precedes it.

We can't change meaning of LEAD-OUT tag because it doesn't represent physical layout of the disc, even if it loads in IsoBuster.

I'm not sure what you mean by that.
Ok, so in the image @tjanas uploaded, the multi-BIN CUE says this: REM LEAD-OUT 01:30:00
Please explain how you interpret this value ?
The way it needs to be interpreted, and the way other implementations interpret this (e.g. ImgBurn), the lead-out starts 1 minute and 30 seconds into FILE "Counting Crows - Hard Candy (Track 13).bin" However, that file (track) is 8 minutes, 45 seconds and 20 frames long. So .. how does that compute ? If an application were to use 01:30:00 it would cut track 13 short, start the lead-out too soon and position track 14 wrong, which is essentially what happens and why other applications can't mount your CUE + multi-BIN
Mind you, in the single-IMG CUE the LEAD-OUT value is correct !

Let me look into this one more time to refresh my knowledge on it. I will get back to you here.
I'm focusing only on Multi-BIN CUE as this is the preservation format we use at redump.

The LEAD-OUT is an address. It is where the lead-out starts, followed by a lead-in and pregap. It is not a size or range, it is an address. The address is relative to the start of the FILE entry that precedes it.

Isobuster uses LEAD-OUT in that way, but redump cue with multi bin is not so.
We had been disscussed in redump forum about the multi session cue. http://forum.redump.org/topic/20409/done-discussing-the-multisession-cue/
If you can't agree to the multi session cue of redump, contact F1ReB4LL and iR0b0t and Jackal, please. If they agree to your opinion, I'll change the LEAD-OUT.

Multi-BIN: I think your understanding of REM LEAD-OUT differs to ours. You assume that REM LEAD-OUT is a total gap size between sessions. We assume that gap between sessions is a sum of 3 sizes: REM LEAD-OUT REM LEAD-IN REM PREGAP

With specifications there really is only one interpretation. The LEAD-OUT is an address. It is where the lead-out starts, followed by a lead-in and pregap. It is not a size or range, it is an address. The address is relative to the start of the FILE entry that precedes it.

We can't change meaning of LEAD-OUT tag because it doesn't represent physical layout of the disc, even if it loads in IsoBuster.

I'm not sure what you mean by that.
Ok, so in the image @tjanas uploaded, the multi-BIN CUE says this: REM LEAD-OUT 01:30:00
Please explain how you interpret this value ?
The way it needs to be interpreted, and the way other implementations interpret this (e.g. ImgBurn), the lead-out starts 1 minute and 30 seconds into FILE "Counting Crows - Hard Candy (Track 13).bin" However, that file (track) is 8 minutes, 45 seconds and 20 frames long. So .. how does that compute ? If an application were to use 01:30:00 it would cut track 13 short, start the lead-out too soon and position track 14 wrong, which is essentially what happens and why other applications can't mount your CUE + multi-BIN
Mind you, in the single-IMG CUE the LEAD-OUT value is correct !

All right, so here is one multi-BIN example:
REM SESSION 01
FILE "Hear It Here First! (USA) (Track 1).bin" BINARY
  TRACK 01 AUDIO
    INDEX 01 00:00:00
FILE "Hear It Here First! (USA) (Track 2).bin" BINARY
  TRACK 02 AUDIO
    INDEX 00 00:00:00
    INDEX 01 00:01:19
FILE "Hear It Here First! (USA) (Track 3).bin" BINARY
  TRACK 03 AUDIO
    INDEX 00 00:00:00
    INDEX 01 00:01:53
REM LEAD-OUT 01:30:00
REM SESSION 02
REM LEAD-IN 01:00:00
REM PREGAP 00:02:00
FILE "Hear It Here First! (USA) (Track 4).bin" BINARY
  TRACK 04 MODE2/2352
    INDEX 01 00:00:00

First of all, data disc Multi-BIN is an invention of redump.org mostly. That's our way of managing the database and there is a lot of sense in this scheme such as matching audio tracks across regions and maintaining a compatibility of reproducing the disc when writing 1:1, with, possibly, write offset deviation, but that's another story. What applies to Single-IMG, doesn't apply to Multi-BIN.
In our case, LEAD-OUT tag is a property of the SESSION, it doesn't "belong" to the last file of the session.
I agree with you that maybe reusing existing CUE tag names to express a different meaning is not a good idea. The discussion on that happened years ago and changing anything at this point will have a huge impact on a lot of things: redump.org DB, user archives, archive.org, rom managers, hardware CD emulators such as PSX PSIO, X-Station, Dreamcast MODE and gdemu.
When it comes to IsoBuster, from what I understand, our users frustration comes mainly from not being able to browse multisession data track contents in your application (because data track is using absolute sector offset as you mentioned earlier).
I have a number of solutions how that can be implemented:

If you see more than one FILE entry in the CUE-sheet - it's redump.org (C) Multi-BIN CUE sheet and you can interpret our tags (LEAD-IN, LEAD-OUT, PREGAP) differently, like I explained in my previous comment.
A variety of (1) where you check if all 3 tags exist (LEAD-IN, LEAD-OUT, PREGAP), if so - it's redump.org (C) Multi-BIN CUE sheet
You don't need CUE-sheet at all. Go over track BIN's, detect if the track is a data track and use first sector MSF as a reference point when you interpret directory records, like here: https://github.com/superg/redump_info/blob/master/src/image_browser.cc#L104

We had been disscussed in redump forum about the multi session cue. http://forum.redump.org/topic/20409/done-discussing-the-multisession-cue/

Interesting, even though I merely scanned the other post just now.  I think @Jackal was on the ball.
If you wanted to do something unique, you should have called it something unique
I also feel the true purpose of LEAD-OUT wasn't really (fully) understood.  It's just an address to show where the LEAD-OUT (and hence also LEAD-IN and PREGAP) starts (since these sizes are fixed per the CD standard).  I wish I had been drawn into that discussion, it could have saved years of confusion.
Water under the bridge

I'll change the LEAD-OUT.

Probably adds even more to the confusion with different flavors in play

changing anything at this point will have a huge impact on a lot of things: redump.org DB, user archives, archive.org, rom managers, hardware CD emulators such as PSX PSIO, X-Station, Dreamcast MODE and gdemu.

Sure !  Not a good idea indeed.

If you see more than one FILE entry in the CUE-sheet - it's redump.org (C) Multi-BIN CUE sheet

Hmm .. No I'm afraid not.  The original CDRWin CUE format supported and documented this type CUE layout.
I also believe other (more in the Audio-CD sphere) apps use multiple files (e.g. *.wav files etc (but *.bin is also perfectly acceptable)).  It's not new nor unique to redump.

A variety of (1) where you check if all 3 tags exist (LEAD-IN, LEAD-OUT, PREGAP), if so - it's redump.org (C) Multi-BIN CUE sheet

I'm not aware of other apps using LEAD-IN but since you used it for years in redump images .. can we be sure other apps have not already adopted it (hopefully in a correct way, namely as an address).
It's worth trying .. but it's thin, I'd like a better check.

You don't need CUE-sheet at all.

Then I could argue: don't create a CUE.
If a user loads a CUE, which is essentially an engineering tool to knit various files together in a CUE-standard-defined way, then the application should follow the CUE and not decide to do things differently than defined in the CUE, because that defeats the engineering power of the CUE.  What is defined in the CUE should be used, no matter the content of the files, so that people can create their own layouts.  That's the spirit of the CUE file essentially.
Frankly I'd much rather like a clear unmistakable TAG in this type CUE that any app can use to recognize the redump flavored multi-BIN + CUE.  Like REM REDUMP MULTI-BIN CUE FORMAT
Or you create two CUES.  The normal one and one with extension *.generic.cue (or something like that) (Perhaps the easiest approach for quick global support)
And likewise you should also ad a new TAG/Command in the CUE that goes together with the CCD+IMG as well.  One that tells an application there is a gap in the IMG.  For instance REM SESSION-GAP 02:32:00.
Missing blocks in an image FILE are allowed but only when clearly defined in the CUE.  PREGAP and POSTGAP are original CDRWin tags/commands that allow this functionality under very strict conditions.  There is no other mechanism to just drop blocks from an image and expect the application to know what was dropped.
E.g. REM SESSION-GAP [MSF] for the combined omitted amount of blocks: lead-out + lead-in + pregap.
The values are known anyways since they are the same on all CDs.  11400 blocks (02:32:00) for the first session transition, 6900 blocks for all latter ones.
All this could be done in a new DIC version without breaking any old functionality.
Third party apps could then start adopting these interpretations because they can clearly detect it and over time confusion would disappear.  At first they would ignore the unknown tags but your user base would alert various tool builders over time I'm sure.

Hmm .. No I'm afraid not. The original CDRWin CUE format supported and documented this type CUE layout. I also believe other (more in the Audio-CD sphere) apps use multiple files (e.g. *.wav files etc (but *.bin is also perfectly acceptable)). It's not new nor unique to redump.
That's irrelevant for for you really, as long as there is no data track in the second session, there are no problems.

Honestly, I don't think discussion will lead anywhere. Based on the tone of your replies it feels like you came not to solve the problem but rather to school us all. Good luck with IsoBuster.

http://forum.redump.org/post/103812/#p103812


Implement better algorithm for changing labels (keep attributes if possible)

My actions before raising this issue

[*] Read/searched the docs
[*] Searched past issues

Expected Behaviour
When changing the type of a label for another with identically set up attributes, the attributes remain the same.
Current Behaviour
When changing the type of a label for another with identically set up attributes, the attributes are reset.
Possible Solution
If label type is changed, check for attributes with identical names and types, and then set the value of the attribute of the new label to that of the old one.
Steps to Reproduce (for bugs)

Create a task with at least two types of labels with identical attributes (ex. name "text" with type "Text")
Create a label on an image and set some value to the text attribute
Change label class, and observe that the attribute is reset

Context

This is a real hindrance for labelling data for OCR training, and there is already a feature request about it (#2755) but no progress on it for months. Also, it is not that "smart" of a feature, it's just a simple heuristic.

It could be a feature request to write a smarter algorithm for changing labels. We would be happy if somebody from the community could prepare a patch. I expect this is client-part only (JavaScript, TypeScript) expertise is necessary.

@timurlenk07 , thanks for your contribution. I will close the issue.


Syntax Error with Additional Configuration

When attempting use the additional configuration to skip creating empty transformation and trigger_table tables I receive the following error:
Running with dbt=0.18.0
Encountered an error:
Runtime Error
  Syntax error near line 38
  ------------------------------
  35 | models:
  36 |   fivetran_log_transformation_status:
  37 |       +enabled: false
  38 |     staging:
  39 |       stg_fivetran_log_trigger_table:
  40 |         +enabled: false
  41 |       stg_fivetran_log_transformation:

  Raw Error:
  ------------------------------
  while parsing a block mapping
    in "<unicode string>", line 36, column 3:
        fivetran_log_transformation_status:
        ^
  expected <block end>, but found '<block mapping start>'
    in "<unicode string>", line 38, column 5:
          staging:
          ^

This is how my dbt_package.yml is currently laid out:
name: sales_engineers

config-version: 2
version: '0.0.1'

# Need to Check With Team To Ensure That Custom Profiles Are Not Supported
profile: fivetran_dbt

require-dbt-version: [">=0.17.0", "<=0.18.0"]

# See Quoting Docs for Snowflake
# https://docs.getdbt.com/reference/project-configs/quoting
quoting:
  database: false
  schema: false
  identifier: false

vars:
  fivetran_log:
      unioning_multiple_destinations: false # disables macro for combining multiple destinations' Log Connectors"
      account: "{{ source('fivetran_log', 'account') }}"
      account_membership: "{{ source('fivetran_log', 'account_membership') }}"
      active_volume: "{{ source('fivetran_log', 'active_volume') }}"
      connector: "{{ source('fivetran_log', 'connector') }}"
      credits_used: "{{ source('fivetran_log', 'credits_used') }}"
      destination: "{{ source('fivetran_log', 'destination') }}"
      destination_membership: "{{ source('fivetran_log', 'destination_membership') }}"
      log: "{{ source('fivetran_log', 'log') }}"
      transformation: "{{ source('fivetran_log', 'transformation') }}"
      trigger_table: "{{ source('fivetran_log', 'trigger_table') }}"
      user: "{{ source('fivetran_log', 'user') }}"

models:
  fivetran_log_transformation_status:
      +enabled: false
    staging:
      stg_fivetran_log_trigger_table:
        +enabled: false
      stg_fivetran_log_transformation:
        +enabled: false

Hi @angel-hernandez91 there are two minor issues here - one is that you need to include fivetran_log as a resource path, and the other is that the alignment was slightly off.  I was able to get the following to run:
models:
  fivetran_log:
    fivetran_log_transformation_status:
      +enabled: false
    staging:
      stg_fivetran_log_trigger_table:
        +enabled: false
      stg_fivetran_log_transformation:
        +enabled: false

I will update the README now. Thanks for raising this problem!

Perfect. It works for me now!


cannot get intrinsics for infrared stream

I'm using using RS415 with FW <IP_ADDRESS>. librealsense 2.8.3, Windows 10 1709.
I'm enabling INFRARED stream and when I try to get instrinsics for this stream I get the message:
19/12 00:54:18,153 WARNING [12036] (types.cpp:57) No intrinsics are available for this stream profile!
I expect the INFRARED stream to have same instrinsics as the depth (which I can get successfully).

seems like its not throwing this error if I set the INFRARED stream to RGB8, but it does with Y16

Hi @svensht2,
For D415 depth camera Infrared Y16 is a calibration (unrectified) format, and does not provide intrinsic by design.
Additionally, not all resolution/fps combination available for Y16 are supported by Z16 depth stream (e.g. FullHD/25fps).

what I really want is Y8, which 415 doesn't like at all, is that expected?

Y8 is supported - you may run RealSense-Viewer and select this format for preview:

Note that in order to get both left and right Infrared streams you need to switch to "Advanced Mode" - see the button "Enable Advanced Mode" in the above image.

I will try using Y8 again

still having the Y8 issue. I'm experiencing the same issue as #937

Tried to run below app (in librealsense 2.8.3) using D415 + FW 5.8.9 and the intrinsics is there. But, the coeffs are all '0'.
$ /rs-enumerate-devices -c

Provided Intrinsic:
Intrinsic of "Depth"	  424x240	  Z16
Width:        	424
Height:       	240
PPX:<PHONE_NUMBER>53711
PPY:<PHONE_NUMBER>40234
Fx:<PHONE_NUMBER>47266
Fy:<PHONE_NUMBER>47266
Distortion:   	Brown Conrady
Coeffs:       	0  	0  	0  	0  	0
Intrinsic of "Depth"	  480x270	  Z16
Width:        	480
Height:       	270
PPX:<PHONE_NUMBER>30273
PPY:<PHONE_NUMBER>09961
Fx:<PHONE_NUMBER>72266
Fy:<PHONE_NUMBER>72266
Distortion:   	Brown Conrady
Coeffs:       	0  	0  	0  	0  	0
Intrinsic of "Infrared 1"	  424x240	  Y8
Width:        	424
Height:       	240
PPX:<PHONE_NUMBER>53711
PPY:<PHONE_NUMBER>40234
Fx:<PHONE_NUMBER>47266
Fy:<PHONE_NUMBER>47266
Distortion:   	Brown Conrady
Coeffs:       	0  	0  	0  	0  	0
Intrinsic of "Infrared 1"	  480x270	  Y8
Width:        	480
Height:       	270
PPX:<PHONE_NUMBER>30273
PPY:<PHONE_NUMBER>09961
Fx:<PHONE_NUMBER>72266
Fy:<PHONE_NUMBER>72266
Distortion:   	Brown Conrady
Coeffs:       	0  	0  	0  	0  	0


fix phpdocs for library and model from CI_Loader

Sorry, but we restrict that to a single type declaration.


Possible to automagically use ${{ github.token }} ?

Currently:

https://github.com/Andrew-Chen-Wang/github-wiki-action/blob/7f44b142379b209235c748351558937d3e7546a7/action.yml#L10-L16
But, I think you can make it magic so you don't need to specify the GH_TOKEN variable:
  GH_TOKEN:
    description: 'The GitHub Token for this action to use. Specify secrets.GITHUB_TOKEN.'
    required: true
    default: ${{ github.token }}
    # 🔺

I don't know 100% for sure, but I think this is possible. I think it's at least worth investigating though.
Here's what I know does work: https://github.com/jcbhmr/publish-wiki-action/blob/a9cd93ca56fd5ed2bf562abc26af3771fb8e63bd/action.yml
runs:
  using: composite
  steps:
    - id: gh-action
      run: '"${GITHUB_ACTION_PATH%/}/index.sh"'
      shell: bash
      # https://github.com/actions/runner/issues/665
      env:
        GITHUB_TOKEN: ${{ github.token }}
        INPUT_PATH: ${{ inputs.path }}

That is pretty interesting, and honestly pretty surprising security wise. Without specified user input of a GitHub token, public repositories (from first glance) are at risk of write vulnerabilities from a third party library/action. Does the above code actually permit pushing without the GitHub action yml file specifying token?

That is pretty interesting, and honestly pretty surprising security wise. Without specified user input of a GitHub token, public repositories (from first glance) are at risk of write vulnerabilities from a third party library/action

I think they were until the default got switched to read-only https://github.blog/changelog/2023-02-02-github-actions-updating-the-default-github_token-permissions-to-read-only/

Does the above code actually permit pushing without the GitHub action yml file specifying token?

This workflow works!
name: Test
on:
  push:
    paths: [index.sh, action.yml, wiki/**, .github/workflows/test.yml]
concurrency: test
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - uses: ./

https://github.com/jcbhmr/publish-wiki-action/blob/a9cd93ca56fd5ed2bf562abc26af3771fb8e63bd/.github/workflows/test.yml
I don't think that it would work if I created a new repo today since the default is contents: read not contents: write (see Changelog blog https://github.blog/changelog/2023-02-02-github-actions-updating-the-default-github_token-permissions-to-read-only/)

Yeah so that repo has inherited the default read-write and I could change it if I wanted

I think doing this implicitly is not great practice, so closing. Feel free to add more comments.

I think doing this implicitly is not great practice, so closing.

To play advocate for the other side, here's some reasons why doing this:
- uses: actions/checkout@v3 # doesn't need GH_TOKEN to pull from repo
- uses: Andrew-Chen-Wang/github-wiki-action # to conform with others, it'd be nice to have default pick it up
- uses: actions/deploy-pages@v1 # picks up token to deploy a website
- uses: stefanzweifel/git-auto-commit-action@v4 # literally pushes to the repo with one line, no token

is preferable to the verbosity of this:
- uses: actions/checkout@v3
  with:
    token: ${{ github.token }}
- uses: Andrew-Chen-Wang/github-wiki-action
  with:
    GH_TOKEN: ${{ github.token }}
- uses: actions/deploy-pages@v1
  with:
    token: ${{ github.token }}
- uses: stefanzweifel/git-auto-commit-action@v4
  with:
    token: ${{ github.token }}

note that you STILL NEED TO DECLARE PERMISSIONS to give that token access to writing the pages, repo, issues, whatever: https://github.com/actions/deploy-pages
    # Grant GITHUB_TOKEN the permissions required to make a Pages deployment
    permissions:
      pages: write      # to deploy to Pages
      id-token: write   # to verify the deployment originates from an appropriate source

Will look into it. I think actually having tests with the CI would help and also having a test on private repos would be great too


Manual set of flatbuffer forwarding target is unavailable

[x] I confirm that this is an issue rather than a question.

Bug report
It is no longer possible to manually set IP of an flatbuffer forwarding destination without mDNS or SSDP.
Steps to reproduce
Run hyperion 2.0.15 or above without ENABLE_MDNS option.
Try to enter IP address in flatbuffer forwarding target in web interface path Network Services > Forwarder > Activate
What is expected?
Available manualy set up hosts and ports for an json and flatbuffer targets, as it works before 2.0.13 release.
What is actually happening?
Error message "Discovery of service type [flatbuffer] via ssdp not supported" occurs.
The hosts and ports parameters fields are still not editable manually.
Forwarding option is unusable without ENABLE_MDNS.
System

Hyperion Server:
- Build:             (HEAD detached at 24a00e3b) (GitHub-70e1043e/24a00e3b-1676809483)
- Build time:        Feb 21 2023 22:31:27
- Git Remote:        https://github.com/hyperion-project/hyperion.ng.git
- Version:           2.0.15
- UI Lang:           en (BrowserLang: ru)
- UI Access:         expert
- Avail Screen Cap.: framebuffer,amlogic
- Avail Video  Cap.: v4l2
- Avail Audio  Cap.:
- Avail Services:    boblight,effectengine,forwarder,flatbuffer,protobuffer,SSDP,borderdetection
- Config path:       /home/osmc/.hyperion
- Database:          read/write

Hyperion Server OS:
- Distribution:      Open Source Media Center
- Architecture:      arm64
- CPU Hardware:      OSMC Vero 4K / Vero 4K +
- Kernel:            linux (4.9.269-44-osmc (WS: 32))
- Root/Admin:        false
- Qt Version:        5.15.2
- Python Version:    3.9.2

As a workaround you can export the configuration, update the forwarder target server with its IP address and then import the config again.
The forwarder process will then use the IP to connect to the remote system.
I am currently at my mobile, but can share the json section to change when being at my PC.

Workaround with export-edit-import works, thanks!
Modified config section:
	"forwarder": {
		"enable": true,
		"flatbuffer": [
			{
				"host": "<IP_ADDRESS>",
				"name": "HyperBian",
				"port": 19400
			}
		],
		"flatbufferselect": [
			"<IP_ADDRESS>"
		],
		"jsonapi": [
			{
				"host": "<IP_ADDRESS>",
				"name": "HyperBian",
				"port": 19444
			}
		],
		"jsonapiselect": [
			"<IP_ADDRESS>"
		]

Good to hear it is working.
Just for completeness, you can also use an DNS-Hostname or an ipv6 address instead of the ipv4 one.


Propagating Coefficient::time [coef-set-time-dev]

The main purpose of this PR is to propagate the Coefficient::time variable into internally stored coefficient objects whenever they exist.
This requires changing Coefficient::SetTime (and analogous functions in related classes) to a virtual function.
This could be classified as a bug fix or an enhancement but since I don't recall any complaints let's call it an enhancement.

This PR is now under review (see the table in the PR description). To help with the review process, please do not force push to the branch.

Many users probably use this, so it may be worth to mention it briefly in CHANGELOG.

@mlstowell, when you get a chance can you address @vladotomov's request?

class MatrixFunctionCoefficient also contains a Coefficient *.

Good catch! Thanks, @vladotomov


[coc.nvim] The TypeScript language service died 5 times right after it got started.

I have been using NeoVim and coc (with extensions) on one "monorepo" for 2-3 months now and everything was working just fine, but since few days/week ago one of the packages that consists of angular application makes my tsserver fail once I open any .ts file within that package. However, it does not fail when I test it on VS Code.
What I have done so far?
NeoVim/coc fixes:

reinstalled coc.nvim
reinstalled all coc extensions (coc-tsserver included)
since I use nvm I have added node path to node version that is being used in coc-settings.json "tsserver.npm": "$HOME/.nvm/versions/node/v16.14.2/bin/npm"

GIT/repo fixes:

deleted repo and cloned again with clean installation
checked out old commit (where tsserver wasn't failing around week ago and even older ones) and rebuild/installed everything cleanly

"Sanity" fixes:

worth mentioning that I have moved from Intel to M1 (MacOS)
however, when I try same fixes (as above) on my old laptop (MacOS Intel), which had tssever working just fine approx week ago - tsserver still fails (tried old commits where things were working smoothly with tsserver, still couldn't make it work)

Error from :CocCommand workspace.showOutput tsserver  is below.
Debug Failure. False expression: position cannot precede the beginning of the file

   Error: Debug Failure. False expression: position cannot precede the beginning of the file
       at computeLineOfPosition ($HOME/.config/coc/extensions/node_modules/coc-tsserver/node_modules/typescript/lib/tsserver.js:9988:22)
       at computeLineAndCharacterOfPosition ($HOME/.config/coc/extensions/node_modules/coc-tsserver/node_modules/typescript/lib/tsserver.js:9966:26)
       at Object.getLineAndCharacterOfPosition ($HOME/.config/coc/extensions/node_modules/coc-tsserver/node_modules/typescript/lib/tsserver.js:10007:16)
       at formatDiagnosticToProtocol ($HOME./config/coc/extensions/node_modules/coc-tsserver/node_modules/typescript/lib/tsserver.js:171642:60)
       at $HOME/.config/coc/extensions/node_modules/coc-tsserver/node_modules/typescript/lib/tsserver.js:172578:93
       at Object.map ($HOME/.config/coc/extensions/node_modules/coc-tsserver/node_modules/typescript/lib/tsserver.js:444:29)
       at IOSession.Session.defaultEventHandler ($HOME/.config/coc/extensions/node_modules/coc-tsserver/node_modules/typescript/lib/tsserver.js:172578:45)
       at ProjectService.eventHandler ($HOME/.config/coc/extensions/node_modules/coc-tsserver/node_modules/typescript/lib/tsserver.js:172484:77)
       at ProjectService.sendConfigFileDiagEvent ($HOME/.config/coc/extensions/node_modules/coc-tsserver/node_modules/typescript/lib/tsserver.js:169830:22)
       at ProjectService.assignProjectToOpenedScriptInfo ($HOME/.config/coc/extensions/node_modules/coc-tsserver/node_modules/typescript/lib/tsserver.js:170669:38)
       at $HOME/.config/coc/extensions/node_modules/coc-tsserver/node_modules/typescript/lib/tsserver.js:170972:97
       at Object.flatMap ($HOME/.config/coc/extensions/node_modules/coc-tsserver/node_modules/typescript/lib/tsserver.js:508:25)
       at ProjectService.applyChangesInOpenFiles ($HOME/.config/coc/extensions/node_modules/coc-tsserver/node_modules/typescript/lib/tsserver.js:170972:41)
       at Session.handlers.ts.Map.ts.getEntries._a.<computed> ($HOME/.config/coc/extensions/node_modules/coc-tsserver/node_modules/typescript/lib/tsserver.js:172131:46)
       at $HOME/.config/coc/extensions/node_modules/coc-tsserver/node_modules/typescript/lib/tsserver.js:174067:88
       at IOSession.Session.executeWithRequestId ($HOME/.config/coc/extensions/node_modules/coc-tsserver/node_modules/typescript/lib/tsserver.js:174058:28)
       at IOSession.Session.executeCommand ($HOME/.config/coc/extensions/node_modules/coc-tsserver/node_modules/typescript/lib/tsserver.js:174067:33)
       at IOSession.Session.onMessage ($HOME/.config/coc/extensions/node_modules/coc-tsserver/node_modules/typescript/lib/tsserver.js:174093:35)
       at Interface.<anonymous> ($HOME/.config/coc/extensions/node_modules/coc-tsserver/node_modules/typescript/lib/tsserver.js:176698:31)
       at Interface.emit (node:events:526:28)
       at Interface._onLine (node:readline:487:10)
       at Interface._normalWrite (node:readline:661:12)
       at Socket.ondata (node:readline:269:10)
       at Socket.emit (node:events:526:28)
       at addChunk (node:internal/streams/readable:315:12)
       at readableAddChunk (node:internal/streams/readable:289:9)
       at Socket.Readable.push (node:internal/streams/readable:228:10)
       at Pipe.onStreamRead (node:internal/stream_base_commons:190:23)

Your tsserver crashed try
  "tsserver.tsdk": "/path/to/typescript/lib",
  "tsserver.ignoreLocalTsserver": true,

to use custom tsserver from typescript module.

Your tsserver crashed ,try configuration:
  "tsserver.tsdk": "/path/to/typescript/lib",
  "tsserver.ignoreLocalTsserver": true,

to use custom tsserver from typescript module.

Tried to set path to local coc-tsserver tsserver module, the module of the package I am using and shared module of the same monorepo. All of them fails.
A little more information that might be helpful from tsserver log
Info 0    [15:13:33.204] Starting TS Server Info 1    [15:13:33.204] Version: 4.6.3 Info 2    [15:13:33.204] Arguments: /Users/oskarasg/.nvm/versions/node/v16.14.2/bin/node /Users/oskarasg/Projects/platform/packages/admin-webapp/node_modules/typescript/lib/tsserver.js --allowLocalPluginLoads --useInferredProjectPerProjectRoot --cancellationPipeName /var/folders/ll/67tkqvv97mgbh2cb9v4fm3xh0000gn/T/nvimrL1N1P/coc.nvim-85520/coc-nvim-tscancellation-594c0526ce9e1d9121d3.sock* --logVerbosity verbose --logFile /var/folders/ll/67tkqvv97mgbh2cb9v4fm3xh0000gn/T/nvimrL1N1P/coc.nvim-85520/tsserver.log --npmLocation "/Users/oskarasg/.nvm/versions/node/v16.14.2/bin/npm" --noGetErrOnBackgroundUpdate --validateDefaultNpmLocation Info 3    [15:13:33.204] Platform: darwin NodeVersion: 16 CaseSensitive: false Info 4    [15:13:33.204] ServerMode: undefined syntaxOnly: false hasUnknownServerMode: undefined Info 5    [15:13:33.207] Binding... Info 6    [15:13:33.211] request:     {"seq":0,"type":"request","command":"configure","arguments":{"hostInfo":"coc-nvim","preferences":{"providePrefixAndSuffixTextForRename":true,"allowRenameOfImportPath":true,"includePackageJsonAutoImports":"auto"}}} Info 7    [15:13:33.212] Host information coc-nvim Info 8    [15:13:33.212] response:     {"seq":0,"type":"response","command":"configure","request_seq":0,"success":true} Perf 9    [15:13:33.212] 0::configure: async elapsed time (in milliseconds) 0.7544 Info 10   [15:13:33.212] request:     {"seq":1,"type":"request","command":"compilerOptionsForInferredProjects","arguments":{"options":{"module":"commonjs","target":"es2016","jsx":"preserve","allowJs":true,"allowSyntheticDefaultImports":true,"allowNonTsExtensions":true}}} Perf 11   [15:13:33.213] 1::compilerOptionsForInferredProjects: elapsed time (in milliseconds) 0.2443 Info 12   [15:13:33.213] response:     {"seq":0,"type":"response","command":"compilerOptionsForInferredProjects","request_seq":1,"success":true,"body":true} Info 13   [15:13:33.213] event:     {"seq":0,"type":"event","event":"typingsInstallerPid","body":{"pid":85562}}

Try a different tsserver version

Try a different tsserver version

Tried 1.10.0, 1.10.1, 1.10.2, 1.10.3, 1.10.4 all of them crashes. Very strange and probably quite stupid problem I am having

Tried 1.10.0, 1.10.1, 1.10.2, 1.10.3, 1.10.4

What are these versions? tsserver version should be same as TypeScript version, 4.6.4 etc.

Got it fixed. It was an Angular "strictTemplates" related issue, just error didn't translate well from tsserver trace.
If anyone encounters anything like that on Angular projects, check this https://github.com/angular/vscode-ng-language-service/issues/1418#issuecomment-1116107353 that is exactly issue I am having. However, it is strange that previously it wouldn't crash Typescript server, but after Angular upgrade to Angular 13 (I suspect) it would.


Optimize environment setup script to speed up test time.

Enable docker cache to speed up container build time.

Coverage remained the same at 7.865% when pulling 6c3145e9ba47c6e1bce5d550738dd43f36c2b7a0 on henrywang:faster into 3a901e09323658c1de9f6fcb665c68f147e5c8c1 on weldr:master.

Coverage remained the same at 7.865% when pulling d477c5d09da7b5e6d2978cadee4015e0d2bf7a87 on henrywang:faster into 3a901e09323658c1de9f6fcb665c68f147e5c8c1 on weldr:master.

Coverage remained the same at 7.865% when pulling deeb4c15350923c3a0bdde48c62bf7a5645b5673 on henrywang:faster into 3a901e09323658c1de9f6fcb665c68f147e5c8c1 on weldr:master.

 Optimize welder-web e2e build and execution times

Coverage remained the same at 7.865% when pulling 40ab960631c8c8639b7cc0656cd2c1d4d7af7f94 on henrywang:faster into 3a901e09323658c1de9f6fcb665c68f147e5c8c1 on weldr:master.

Coverage remained the same at 7.865% when pulling e70f3bbac1a0b072338b3da22a0acc404ecd7c59 on henrywang:faster into 3a901e09323658c1de9f6fcb665c68f147e5c8c1 on weldr:master.

Coverage remained the same at 7.865% when pulling f1de0c8498f26b6955f4d38b38c2704caf74a06b on henrywang:faster into 3a901e09323658c1de9f6fcb665c68f147e5c8c1 on weldr:master.

Coverage remained the same at 7.865% when pulling 7238c45cb583abed529b16422aa29d49d39719e6 on henrywang:faster into 3a901e09323658c1de9f6fcb665c68f147e5c8c1 on weldr:master.

Coverage remained the same at 7.865% when pulling 16ecd40d61325cfbba0ad0d4c3e3b2e9b5def77b on henrywang:faster into 3a901e09323658c1de9f6fcb665c68f147e5c8c1 on weldr:master.

Coverage remained the same at 7.865% when pulling 189e21a9c4b3a312a95e52f17027d2740a26e6ea on henrywang:faster into 3a901e09323658c1de9f6fcb665c68f147e5c8c1 on weldr:master.

Coverage remained the same at 7.881% when pulling 7090a9eac069d1db82008c49351de5b973998fb6 on henrywang:faster into f1980b34380914e3fc11e31bff006e8b932229f1 on weldr:master.

Coverage remained the same at 7.881% when pulling da92cf824818987b9409d15c12c072d5e6f82405 on henrywang:faster into f1980b34380914e3fc11e31bff006e8b932229f1 on weldr:master.

Coverage remained the same at 7.881% when pulling 74ad7542fb24bfd6028a818466a7940d6a9b5003 on henrywang:faster into f1980b34380914e3fc11e31bff006e8b932229f1 on weldr:master.

Coverage remained the same at 7.881% when pulling b5f8ebf44fb60c26b37078f77d76c4d9bbb4955f on henrywang:faster into f1980b34380914e3fc11e31bff006e8b932229f1 on weldr:master.

Coverage remained the same at 7.881% when pulling cc67946beccdb6a0940aaf3c2554344aa6e8cf1b on henrywang:faster into f1980b34380914e3fc11e31bff006e8b932229f1 on weldr:master.

Closing in favor of #132


CupertinoNavigationBarBackButton fires error when used with NavigationService

I have an app created like so:
@StackedApp(routes: [
  CupertinoRoute(page: AppView, initial: true),
  CupertinoRoute(page: PhoneNumberView),
  CupertinoRoute(page: VerificationCodeView),
])
class App extends StatelessWidget {
  const App({super.key});

  @override
  Widget build(BuildContext context) {
    return MediaQuery.fromWindow(
      child: CupertinoApp(
        useInheritedMediaQuery: true,
        theme: const CupertinoThemeData(),
        navigatorObservers: [StackedService.routeObserver],
        navigatorKey: StackedService.navigatorKey,
        initialRoute: Routes.appView,
        onGenerateRoute: StackedRouter().onGenerateRoute,
      ),
    );
  }
}

And a view that contains a navigator bar created like this:
return ViewModelBuilder<AuthenticationViewModel>.nonReactive(
      viewModelBuilder: () => getIt.get<AuthenticationViewModel>(),
      builder: (context, model, child) => CupertinoPageScaffold(
        navigationBar: CupertinoNavigationBar(
          backgroundColor: CupertinoColors.systemBackground,
          border: null,
          padding: const EdgeInsetsDirectional.all(0),
          leading: CupertinoNavigationBarBackButton(
            color: CupertinoColors.label,
            // if I add this it works, but would prefer it work by default rather than using a custom widget
            // onPressed: () => Navigator.pop(context),
          ),
        ),
        child: ...

Yet when I try to navigate back / pop with a CupertinoNavigationBarBackButton, it throws this error:
CupertinoNavigationBarBackButton should only be used in routes that can be popped
'package:flutter/src/cupertino/nav_bar.dart':
package:flutter/…/cupertino/nav_bar.dart:1
Failed assertion: line 1336 pos 9: 'currentRoute?.canPop ?? false'

I understand that if I attach a custom action to the back button it will no longer throw this error and perhaps that's the easiest fix, but I am wondering if there is anything that can be done to get this to work by default without throwing an error. I though adding the navigatorObserver in my app would resolve the issue, but that didn't seem to work.

Hi, Thank you for the detailed issue. I appreciate it.
Very interesting, I actually have never used that back button before. I always call either .pop or NavigationService.back depending on what my ViewModel inherited.
I'll leave this open and we'll tend to it when some higher-priority items are completed.
@ferrarafer you have any idea why this could be happening or why it's not picking up that there is a route to pop?

@bryan-vh and @FilledStacks
This error is not related to Stacked. See Flutter #89888
Your only option is to completly avoid setting the CupertinoNavigationBarBackButton or the page on the stack before the CuppertinoScaffold has to be a material Scaffold. Then you can also use the custom back button.

Thanks @ynnob you're awesome. Closing this because it's a Flutter issue


Firefox extension master

Commentary please. What is this?

I don't think you intended to open this for review, so I am closing it. Feel free to comment otherwise.


terragrunt: 0.29.4 -> 0.29.8

Motivation for this change
Bump terragrunt to 0.29.8
Things done

[X] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[X] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[X] Tested execution of all binary files (usually in ./result/bin/)
[ ] Added a release notes entry if the change is major or breaking
[X] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 125676 run on x86_64-linux 1

1 package built:

terragrunt


TNC does not return optimal parameters

The truncated Newton (TNC) optimizer scipy.optimize.minimize(method='TNC') returns apparently not the optimal parameter vector and function value, but the last one, which often (depending on the dimension of the problem in 10-30% of cases in our applications) has a higher function value.
Expected behavior: It should return the optimal found parameters and function value.
This behavior was not observed for any other scipy optimizer considered.
Reproducing code example:
We encountered this problem in our parameter estimation tool github.com/icb-dcm/pypesto, which employs an objective function which records a history of all function evaluations and can thus report if the best value in the trace is better than the value reported by the optimizer. See here for a discussion: https://github.com/ICB-DCM/pyPESTO/issues/327. The error can be reproduced in cell 5 of the notebook https://github.com/ICB-DCM/pyPESTO/blob/master/doc/example/rosenbrock.ipynb. The problem only has box constraints, which are not violated.
Error message:
Function values from history and optimizer do not match: 1.3168387678656086, 2.0179911928533514
Parameters obtained from history and optimizer do not match: [0.99553374 0.98995083 0.97544456 0.94768476 0.89550744 0.8019584
 0.64185456 0.40180156 0.14794723 0.00844711], [0.9940831  0.98497745 0.96635783 0.93780681 0.87380595 0.74153588
 0.53395112 0.25523609 0.03957006 0.00100739]
Function values from history and optimizer do not match: 1.6349167203078765, 2.336760892779954

Scipy/Numpy/Python version information:
1.4.1 / 1.18.4 / 3.7.4

Case that demonstrates:
import numpy as np
from scipy.optimize import rosen, minimize

class Func():
    def __init__(self):
        self.nfev = 0
        self.best_x = None
        self.best_f = np.inf

    def __call__(self, x, *args, **kwds):
        f = rosen(x)
        self.nfev += 1
        if f < self.best_f:
            self.best_f = f
            self.best_x = x
        return f

f = Func()
res = minimize(f, [1, 1.1, 1.2, 1.3, 1.4], method='TNC', bounds=[(-5, 5)] * 5)

assert f.nfev == res.nfev
np.testing.assert_equal(f.best_f, res.fun)
np.testing.assert_equal(f.best_x, res.x)

gives
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-13-0112ed94cec0> in <module>
     20
     21 assert f.nfev == res.nfev
---> 22 np.testing.assert_equal(f.best_f, res.fun)
     23 np.testing.assert_equal(f.best_x, res.x)

~/miniconda3/envs/dev3/lib/python3.8/site-packages/numpy/testing/_private/utils.py in assert_equal(actual, desired, err_msg, verbose)
    426         # Explicitly use __eq__ for comparison, gh-2552
    427         if not (desired == actual):
--> 428             raise AssertionError(msg)
    429
    430     except (DeprecationWarning, FutureWarning) as e:

AssertionError:
Items are not equal:
 ACTUAL: 2.538427471052184e-07
 DESIRED: 2.5384565042871764e-07

In the case of TNC there is a big discrepancy. It's obviously a long way from the minimum ([1, 1, 1, 1, 1]). In this particular example 'TNC' reaches the maximum number of iterations. With more iterations than the default we get a lot closer to the True minimum and the discrepancy gets much less (9.625356727615683e-10 vs 9.606951571011453e-10). However, they're still not the same.
The best_fun is probably done by an intermediate function evaluation (e.g. during line search, or gradient evaluation), and is not the same as the best value being tracked by the iteration loop in the minimizer. Ideally the two would be the same. That particular kind of issue would probably pop up for a lot of minimizers. I'm not sure if one should classify that as a bug because those techniques typically don't keep track of the best function value in the way our example uses.
If, however, there is a stale value not being updated in a loop somewhere, then that's a more serious issue.
One possible solution in this situation is to add best_fun and best_x attributes to ScalarFunction. Pretty much all the minimizers go through this object, and it would be easy to cache the best solution evaluated so far (and check it against the final solution). However, I'm not sure how desirable this behaviour is. One would have to do checks against all the constraints, and whether the bounds were satisfied, and the returned jac and hess values might not sync exactly with the x location.

Hi @andyfaff , thanks for creating the case demonstration. This seems to be the exact problem. I agree that just caching the best solution inside ScalarFunction may lead to problems as some routines may evaluate points that do not satisfy the constraints. If one implements it this way, one would need to 1) check that all constraints are satisfied, and 2) check if the gradient etc. are right. One could perform one additional gradient evaluation at the end if in doubt or return None. Additional problems may come up if a Hessian approximation is used, which needs to be sequentially refined over the optimizer course and thus cannot be reproduced from a single point.
Since we did not observe this behavior for any other optimizers (we had one possible case with L-BFGS, but I cannot reproduce that, so I'm not sure), it might be specific to TNC and there indeed a weird behavior of the algorithm, or a bug, as you point out.

So lets have a look at all the minimizers:
import numpy as np
from scipy.optimize import rosen, minimize, rosen_der, rosen_hess

class F():
    def __init__(self):
        self.best_fun = np.inf
        self.best_x = None

    def __call__(self, x):
        fun = rosen(x)
        if fun < self.best_fun:
            self.best_fun = fun
            self.best_x = x
        return fun

meths = ['nelder-mead', 'cg', 'bfgs', 'l-bfgs-b', 'tnc', 'cobyla', 'slsqp', 'trust-constr',
 'dogleg', 'trust-ncg', 'trust-exact', 'trust-krylov']

for method in meths:
    jac = hess = None
    if method in ('newton-cg', 'trust-krylov', 'trust-exact',
                  'trust-ncg', 'dogleg'):
        jac = rosen_der
    if method in ('trust-krylov', 'trust-exact', 'trust-ncg',
                  'dogleg'):
        hess = rosen_hess

    f = F()
    res = minimize(f, [1, -2, 3, 4, 5], method=method, jac=jac, hess=hess)
    print(method, np.equal(res.fun, f.best_fun), res.fun, f.best_fun, res.success, res.message)

gives
nelder-mead True 0.6841550038341955 0.6841550038341955 False Maximum number of function evaluations has been exceeded.
cg False 1.944273508185588e-10 1.9431177708679375e-10 True Optimization terminated successfully.
bfgs False 4.609168005831311e-11 4.437134484491673e-11 False Desired error not necessarily achieved due to precision loss.
l-bfgs-b False 2.2490199818718852e-11 2.2321289184993123e-11 True b'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH'
tnc False 2.7271265087711303 2.4253707839139977 False Max. number of function evaluations reached
cobyla True<PHONE_NUMBER>8002674<PHONE_NUMBER>8002674 False Maximum number of function evaluations has been exceeded.
slsqp False 1.563858402399705e-08 1.5589201215269426e-08 True Optimization terminated successfully.
trust-constr False 5.143020768594074e-11 5.1369046800844246e-11 True `xtol` termination condition is satisfied.
dogleg True 11.718733542698956 11.718733542698956 False A linalg error occurred, such as a non-psd Hessian.
trust-ncg True 2.190618908717441e-15 2.190618908717441e-15 True Optimization terminated successfully.
trust-exact True 9.449221937588392e-16 9.449221937588392e-16 True Optimization terminated successfully.
trust-krylov True 1.549679356983471e-17 1.549679356983471e-17 True Optimization terminated successfully.

You'll see that the minimisation with TNC does not result in OptimizeResult.success being True. i.e. the minimisation failed. Ostensibly several other methods failed as well, mainly because iteration limits were reached. You definitely need to be checking the success attribute. I suspect in your examples you weren't.
Now let's revisit the minimum function value being visited by the objective. Imagine a solver whose mode of operation is fun(x) --> jac(x) --> fun(x).
Lets say on the second fun(x) evaluation (fval2) that the amount of fval decrease over the first fun(x) (fval1) has reached the criterion for stopping. At this point you'd expect the OptimizeResult.fun value to be the same as the second fval2.  However, calculation of jac(x) uses finite differences (with many fun(x) evaluations). It may be that some of those  FD evaluations may be at slightly lower fval than the second fval2.  In that case the right thing to do is still return fval2, it's not a bug, rather the way that the solvers work. One such example is the l-bfgs-b above, for which OptimizerResult == True, but for which OptimizerResult.fun is slightly larger than that observed in the trace. The way to get to lower function values is to set stopping tolerances much tighter.
What would be a bug would be the optimizer halting based on the process I've just described, but with the OptimizeResult.fun being equal to fval1.
As such, I don't think the OP is evidence for an actual bug in the solvers.

Hi, thanks for the profound analysis.

You'll see that the minimisation with TNC does not result in OptimizeResult.success being True. i.e. the minimisation failed. Ostensibly several other methods failed as well, mainly because iteration limits were reached. You definitely need to be checking the success attribute. I suspect in your examples you weren't.

Yes, we did not check success as we employ a multi-start optimization where we usually have to live with whatever each of 100s of single starts returns. Of course makes sense to check, maybe we'll implement that for promising points, but just to clarify.

However, calculation of jac(x) uses finite differences (with many fun(x) evaluations). It may be that some of those FD evaluations may be at slightly lower fval than the second fval2. In that case the right thing to do is still return fval2, it's not a bug, rather the way that the solvers work.

I see your point about FD. Since we provided analytic gradients (which you did not for all optimizers above), I suspect far less optimizers showed this behavior than above, except some which evaluated points which were then not used to update the trace history, like, potentially, TNC. I agree that returning fval2 is the "right thing to do" from the search algorithm perspective. If one is just interested in overall best parameter values, it is of course always possible to manually self-record them, as in Func above, which we also do.

As such, I don't think the OP is evidence for an actual bug in the solvers.

Agreed. It was striking that TNC showed this behavior quite often compared to the others (providing gradients), and seems to always return the last evaluated function value, not the best, the difference being beyond any tolerance threshold. Whether this is a flaw of the algorithm (corresponding to returning fval1, not fval2) or not this use case fails to reveal, this I guess only a code analysis could clarify.

Here's for completion an illustration of how only TNC fails in gradient-based mode:
import numpy as np
from scipy.optimize import rosen, minimize, rosen_der, rosen_hess

class F():
    def __init__(self):
        self.best_fun = np.inf
        self.best_x = None

    def __call__(self, x):
        fun = rosen(x)
        if fun < self.best_fun:
            self.best_fun = fun
            self.best_x = x
        return fun

meths = ['nelder-mead', 'cg', 'bfgs', 'l-bfgs-b', 'tnc', 'cobyla', 'slsqp', 'trust-constr',
 'dogleg', 'trust-ncg', 'trust-exact', 'trust-krylov']

for method in meths:
    jac = hess = None
    #if method in ('newton-cg', 'trust-krylov', 'trust-exact',
    #              'trust-ncg', 'dogleg'):
    jac = rosen_der
    #if method in ('trust-krylov', 'trust-exact', 'trust-ncg',
    #              'dogleg'):
    hess = rosen_hess
    x0 = [-0.16584467,  1.31897148, -3.26098005,  3.45927947,  0.29453825,
        1.31031958, -1.75957634, -1.83706183,  3.42357846, -0.78687309]
    f = F()
    res = minimize(f, x0, method=method, jac=jac, hess=hess, options={'maxiter': 100})
    print(method, np.equal(res.fun, f.best_fun), res.fun, f.best_fun, res.success, res.message)

gives
nelder-mead True 2120.269071102054 2120.269071102054 False Maximum number of iterations has been exceeded.
cg True 0.26533809379430673 0.26533809379430673 False Maximum number of iterations has been exceeded.
bfgs True 2.5370442739135357e-14 2.5370442739135357e-14 False Maximum number of iterations has been exceeded.
l-bfgs-b True 2.7822653573972946e-09 2.7822653573972946e-09 True b'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH'
tnc False 4.755755190559484 3.987300664638279 False Max. number of function evaluations reached
cobyla True 69.73757558350557 69.73757558350557 False Maximum number of function evaluations has been exceeded.
slsqp True 2.6276887439165257 2.6276887439165257 False Iteration limit exceeded
trust-constr True 8.597660362768444e-21 8.597660362768444e-21 True `gtol` termination condition is satisfied.
dogleg True 40923.23016898274 40923.23016898274 False A linalg error occurred, such as a non-psd Hessian.
trust-ncg True 7.202214306674385e-21 7.202214306674385e-21 True Optimization terminated successfully.
trust-exact True 6.761705476594788e-16 6.761705476594788e-16 True Optimization terminated successfully.
trust-krylov True 4.911506820708559e-13 4.911506820708559e-13 True Optimization terminated successfully.

Indeed, setting maxiter=1000, the found values match (True) for all optimizers, indicating this may only occur when convergence was not reached.

Closing because I think the following is True:

However, calculation of jac(x) uses finite differences (with many fun(x) evaluations). It may be that some of those FD evaluations (or evaluations in linesearch) may be at slightly lower fval than the second fval2.

Note that some of the examples had res.success=False because of hitting the maxfun limit. If this is happening it's not a True demonstration of the issue; set maxfun higher and repeat.
Furthermore, there is an issue when using TNC with a callback function, the callback mechanism in the TNC code was mutating the output. This is being fixed in #14882.


[bug] Async streaming validation duplicates output in the presence of multiple validators

Quick thank you to user new in the Discord.  Link to the thread: https://discord.com/channels/1085077079697150023/1288085320805388298/1288085864521666591
Describe the bug
When AsyncGuard.use_many is given multiple guards (in this case, DetectPII and ToxicLanguage) with on_fail="fix", the stream of outputs will be reduplicated.
To Reproduce
Steps to reproduce the behavior:
import asyncio
import os
from dotenv import load_dotenv
import litellm
import openai
import guardrails

from guardrails.hub import DetectPII, ToxicLanguage

from typing import Callable, Dict, Optional
from guardrails.validators import (
    PassResult,
    register_validator,
    ValidationResult,
    Validator,
)

# Load environment variables
load_dotenv()
openai.api_base = os.getenv("OPENAI_API_BASE")
openai.api_key = os.getenv("OPENAI_API_KEY")

# VERSION 1
guard = guardrails.AsyncGuard().use_many(
    DetectPII(pii_entities="pii", on_fail="fix")
)

# VERSION 2
# guard = guardrails.AsyncGuard().use_many(
#    DetectPII(pii_entities="pii", on_fail="fix"), ToxicLanguage(on_fail="fix")
# )

async def generate_text():
    fragment_generator = await guard(
            litellm.acompletion,
            api_key=openai.api_key,
            api_base=openai.api_base,
            model="openai/mistralai/Mistral-Nemo-Instruct-2407",
            messages=[
                {"role": "system", "content": "Only write my sentences provided please and nothing else please."},
                {
                    "role": "user",
                    "content": """Peter is funny and lives in New York. My name is Peter. Who are you Brian ?""",
                },
            ],
            max_tokens=1024,
            temperature=0,
            stream=True,
        )

    text = ""
    async for op in fragment_generator:
            print(op)
            await asyncio.sleep(0)
            text += op.validated_output

    print(text)
# Run the async function to generate text
import asyncio

asyncio.run(generate_text())

Expected behavior
Example model output: "My friend Alex is a researcher at Purdue University. (Numerous Obscenities)"
Expected cleaned output: "My friend  is a researcher at ."
Observed output: "My friend My friend friend  is a researcher at friend  is a researcher at ."
Library version:
Guardrails 0.5.10
Additional context
Happens in a notebook and in a terminal.

CC @nichwch I think your async changes fixed this, right?


Maximum crime in which election

I have visualized and made a generalized model for maximum crime in which year of election.
Solution to #22 @souvikg544

@dhruvkej9 Merged your solution


Shrinkstack() exception

Just opening this so it isn't lost in Discord.
See: https://discord.com/channels/847318834952405042/847318834952405045/943703323691876453

This is caused by ||= being used to assign to a dereference.
Works: some_var ||= list()
Broke: some_obj.some_var ||= list()

As of 70953a678c14f0d9df21603fe2086cc026ba4619 this is no longer an exception. It's now a compiler error.


Update readme

This PR aims to update Readme files to provide a better introduction of the rapids-js modules

Looks great!


Release for Vaadin 24

Hello,
are you planning release for Vaadin 24?
I see commits with 24 support but it was not released yet, was it?
Thx
JS

Hello. The current version (2.1.1) is already compatible with Vaadin 14 to 24 (demo at https://addonsv24.flowingcode.com/simple-timer/simple-timer). There was no need of releasing a new major version of the add-on.

Oh, ok then. I'm closing this issue. Thank you🙂


fixes #121: initial commit

fixes #121
fixes #164
fixes #157
fixes #48

Outstanding issues to be addressed in this PR:

Issue with merging, should clone values. Otherwise extends from a file listed on CLI may use intermediate result of merging files instead of purely the file
simplify structs (#211) to make coding easier
finish implementation of New function in pkg/docker/compose/config (merging services and merging of XProperties from different files)
behavior when extending from different version? raise issue
behavior when merging different version of files? raise issue
update README.md (limitations particularly)


Bug: PSO: incorrect velocity calculation after simplification update

With the removal of the neighbourhood topology a slight mistake was made:
const arma::Col<double>& velocity =           maximalAcceleration_ * arma::randu<arma::Col<double>>(numberOfDimensions_) % velocities_.col(activeParticleIndex_) + randomNeighbour(attractionCenter, 0, arma::norm(attractionCenter));
was simplified into
const arma::vec& velocity = randomNeighbour(getMaximalAcceleration() * mant::randu(numberOfDimensions_) % velocities_.col(n), 0, arma::norm(attractionCenter));
which apparently is wrong (not even 30% success rate on sphere 5D vs >99% with old codeline). I suspect it's because randomNeighbour(...) is supposed to be added ontop of the calculation there.

This bug still applies, despite something being wrong with mant::randu right now.

Thanks for the catch! The problem seems to be the missing attractionCenter als Position in the second case.
m = maximalAcceleration_
u = arma::randu<arma::Col<double>>(numberOfDimensions_)
v = velocities_.col(activeParticleIndex_)
a = attractionCenter
.*= element-wise multiplication
First case: m * u .* v + randomNeighbour(a, 0, |a|)
Erroneous second case: randomNeighbour(m * u .* v, 0, |a|)
Corrected second case: randomNeighbour(m * u .* v + a, 0, |a|)
However, I think the first case is more focused on the calculation this was derived from.

This was also resolved by #214


VesSAP data Preprocess

I couldn't find the VesSAP data preprocess
Please can you tell me the place of preprocess VesSAP data code

Hi, the preprocessing happens in the feats.py after the prediction: https://github.com/vessap/vessap/blob/02cf71167c740a17645c635bb6e171b655457425/feats.py#L91
Cheers!


[2023 Yearly Social Media ] Call to action:  MPWG Prospectus highlight - Become a Member

Yearly Spreadsheet Media Inventory
Media Description
Becoming a member of the MicroProfile Working Group demonstrates a commitment to the evolution and sustainability of #MicroProfile while ensuring the continued growth and development of a well-governed, vendor-neutral #OpenSource ecosystem.
Join us: https://microprofile.io/workinggroup/#prospectus
Schedule
1 month
2022 issue
https://github.com/eclipse/microprofile-marketing/issues/664

Scheduled


Add @Schema annotation to SingularityScheduledTasksInfo for JSON parsing

Corrects https://github.com/HubSpot/Singularity/pull/2260
https://git.hubteam.com/HubSpot/PaaS-Run/issues/1588

🚢


Add extension methods to IEnumerable<OptionsDescriptor>

It is currently rather cumbersome to configure options in any of the lists in MvcOptions. The user has to write selector code to get to the descriptor objects, and then get the actual instance. For example
services.ConfigureOptions<MvcOptions>(options =>
    options.InputFormatters.Single(fd => fd.Instance is JsonInputFormatter).Instance.XXXX

Instead we should have a set of extension methods allowing easy access to the descriptors/instances
Here is an initial suggestion - @rynowak lets discuss and update
public static TInstance InstanceOf<TInstance>(this IEnumerable<OptionsDescriptor<TInstnace>> collection);
public static TInstance InstanceOfOrDefault<TInstance>(this IEnumerable<OptionsDescriptor<TInstnace>> collection);
public static IEnumerable<TInstance> InstancesOf<TInstance>(this IEnumerable<OptionsDescriptor<TInstnace>> collection);
public static void RemoveTypesOf<TDescriptor>(this IList<OptionsDescriptor<TDescriptor>> collection);

perhaps we should also consider a replace in place extension, so a customized options can come in the exact same spot the one it replaced is in.
#1744

I'm ready to do this one now, and I can submit a PR for it.

It looks like extensions to IEnumerable<OptionDescriptor> aren't going to provide the desired result.  For example this method:
public static TOption InstanceOf<TOption>([NotNull] this IEnumerable<OptionDescriptor<TOption>> descriptors)

Called by this code:
var formatter = new MvcOptions().InputFormatters.InstanceOf<JsonInputFormatter>();

Will produce a CS1929 compilation error: The extension method requires a receiver of type IEnumerable<OptionDescriptor<JsonInputFormatter>>.
I would like to suggest the following (for both input and output descriptors):
public static InputFormatterDescriptor InstanceOf<TOption>([NotNull] this IEnumerable<InputFormatterDescriptor> descriptors)

You can alternatively define the IOptionsDescriptor<T> interface and use InstanceOf<T>(IEnumerable<object>)
e.g.
public class InputFormatterDescriptor : OptionDescriptor<IInputFormatter>,
                   IOptionDescriptor<IInputFormatter>
{
     ...
}

and
public interface IOptionDescriptor<out TOption>
{
    Type OptionType { get; }
    TOption Instance { get; }
}

and
public class OptionDescriptor<TOption> : IOptionDescriptor<TOption>
{
    ....
}

and finally
public static TInstance InstanceOf<TInstance>([NotNull] this IEnumerable<IOptionDescriptor<object>> descriptors)
{
    return default(TInstance);
}

Perfect ... this will work for InstanceOf, InstanceOfOrDefault  and InstancesOf.  However, RemoveTypesOf  becomes a bit problematic, because IList is not covariant.  For example, this method:
public static void RemoveTypesOf<TInstance>([NotNull] this
    IList<IOptionDescriptor<object>> descriptors)

Will result in a compilation error when called by this method:
new MvcOptions().InputFormatters.RemoveTypesOf<JsonInputFormatter>();

Because it can't convert from InputFormatterDescriptor to IOptionDescriptor<object>.
Instead, RemoveTypesOf will need to be defined like so:
public static void RemoveTypesOf<TInstance, TOptionDescriptor>([NotNull] this
    IList<TOptionDescriptor> descriptors)
    where TOptionDescriptor : IOptionDescriptor<object>

And it will need to be called with an extra type argument, like so:
new MvcOptions().InputFormatters.RemoveTypesOf<JsonInputFormatter,
    InputFormatterDescriptor>();

Which is not very intuitive.  Would you like me to add a RemoveTypesOf method anyway?

Fixed with commits 2c21c7c9c7e83a058eb5aab7364119b52d2bc833 and 45aae5956d83006fdcba29cd464bb63d99323944.


TOOL: Application Tab | Remove Application ID column and change Application name column to give more info

As a user, the application id column does not give me any useful information. The application name column should also so "Telework Agreement for XXXXX"/ SL Review form for XXXXXXX", depending on a specific field in particular to that form.
Tasks

[ ] Remove Application ID column

[ ] Change/add new cloumn to say  "Telework Agreement for XXXXX"/ SL Review form for XXXXXXX"

[ ] Change name of columns where it say application to "Form"

For telework: field name (employee name)
For SL Review: field name (enter employee name)
Acceptance criteria
When the required info for submitted applications are available in the applications tab

@bhumin-fw the name of the SL review 2.0 form was changed to "Senior Leadership Review Form". The mapping of the senior leader's name needs to go with this new updated name. Giving you a heads-up. Also, I posted a similar comment for the drafts tab ticket

@HasanSifat It is ready to test on DEV. Fix name not display issue in Application and Draft both place.

This looks good now. One more issue has come up, the new entries for the Senior Leadership Review Form For XXXXXX, the edit button is not available. Only the view button is available. The edit feature needs to be there for users to go back and edit the form. Screenshot for reference-

This might be happening because of the name change of the form and certain parameters in the Camunda workflow not being mapped to the correct name of the form.

@HasanSifat I'm looking into it. Seems like there a connection issue between Camunda and Formio.

tested successfully in DEV. Ready to be transferred to TEST/PROD

Approved - ready to move to test

Tested successfully in TEST

successfully tested in PROD. Closing ticket


Fix with typing effect

Typing effect is glitching on shell
What happens
When run on shell, the program waits a while and then prints text.

Fix: Fix Shell Bug
Don't worry, there there will be more issues...


Improve the chart type of Visualize in sqllab

Improve the chart type of Visualize in sqllab
Fix the style of the list page when there is no data
Fix the link address in the navbar

Coverage remained the same at 69.34% when pulling f0c9460311f51b8a0b1adcf48805720eb5be23a7 on eeve:master into 4c3313b01cb508ced8519a68f6479db423974929 on apache:master.

Many of the charts have requirements that won't be fulfilled when landing on the "explore view". I'd rather support only a subset of the charts in the "visualize flow" and make sure that we fill in the values needed for the controls for them to render properly as you land.
Your approach in building the dropdown is better though. Can you add a .filter() that specifies a subset of the vizTypes?
Some of the controls like metrics will have pick a default value, but some required ones won't. Eventually we should make sure that all required field get a decent default so that the visualize flow always works.

Glad to see you're working on this. There was some issues in SQL Lab around resizing the window where the height of some components wouldn't be altered as they should. Please make sure resizing the window work well along with this set of changes.

Coverage decreased (-0.03%) to 69.311% when pulling d6ad1e01753a150a2c3fb0e966c5f5c1b7b50ea0 on eeve:master into 4c3313b01cb508ced8519a68f6479db423974929 on apache:master.


[WIP] testing travis with non-fork/branch PR

TESTING ONLY.
This is the local branch version of this change. I'm using it to test the latest changes on travis.

closing, the test was successful, integration tests ran. See #517 for the real PR from my fork.


Giving staff an indication before an email is sent to the applicant

Staff are frequently confused about when applicants are sent an email notification about a change/transition made to their application. A small warning on the Update Status prompt when a status is selected that will notify the applicant would be helpful. Maybe requiring some HTMX in that modal?

Why does it matter when a status update goes out? I can see why an organisation would like to turn it off globally but not why staff should be bother by it.