Support encrypted synchronized Realms

Still missing changelog entries for this and
other changes that happen to have been pulled in due to the
core, sync and object store updates

Hmm...tests are failing with fatal error: 'sync_config.hpp' file not found: #include "sync_config.hpp". Can you check the target membership settings on the object store files in Xcode?

I've added some changelog entries. I don't think realm/realm-core#2415 was ever triggerable by our end users, so I didn't include that in the changelog entries, but I'd appreciate if someone could confirm.

realm/realm-core#2415 is only relevant with a true dynamic schema, which we don't support.

Addressed all feedback so far, have changelog entries, updated to core 2.5.1 and sync 1.5.0. Object store is at realm/realm-object-store#420, which I'll update to point to master once that PR passes CI and is merged.


PixelSpacing incorrectly taken into account

Hi all,
Thanks for all your work, rt-utils is a great tool !
I am working with a MR image and sometimes, depending of the aquisitiion  we need to apply a rotation so that the images are "axialized".
The result is that the pixelSpacing has a different value for x and y and the rtstruct generated is incorrect.
Here is a python script that allows to create a SEG and a RT
Please set force=True in rt_utils when loading the dcm with pydicom, I had to create dcm file to have a self content script
import glob
import os
import pydicom_seg
import pydicom
import SimpleITK as sitk
import numpy as np
from rt_utils import RTStructBuilder
import json

# To update based on the path
dcm_dir = "C:\\Temp\\dicom_file_new"
seg_filename = "C:\\Temp\\segmentation.dcm"
rt_filename = "C:\\Temp\\segmentationRT.dcm"
template_filename = "C:\\Temp\\template.json"
isotrop = False

def process_image(image_data):
    out = np.where(image_data == 180, 1, 0).astype(np.uint16)
    return out

def create_input_dicom(dicom_dir, isotrop):
    os.makedirs(dicom_dir, exist_ok=True)

    header = '{"00080005": {"Value": ["ISO_IR 192"], "vr": "CS"}, "00080008": {"vr": "CS"}, "00080012": {"Value": ["20230721"], "vr": "DA"}, "00080013": {"Value": ["035751"], "vr": "TM"}, "00080014": {"Value": ["3.0.0"], "vr": "UI"}, "00080016": {"Value": ["1.2.840.10<IP_ADDRESS>.1.1.4"], "vr": "UI"}, "00080018": {"Value": ["2.25.115231125129161360422423700315339076734"], "vr": "UI"}, "00080020": {"Value": ["20150516"], "vr": "DA"}, "00080021": {"Value": ["20230721"], "vr": "DA"}, "00080022": {"Value": ["20150516"], "vr": "DA"}, "00080023": {"Value": ["20230721"], "vr": "DA"}, "00080030": {"vr": "TM"}, "00080031": {"Value": ["035751"], "vr": "TM"}, "00080032": {"vr": "TM"}, "00080033": {"Value": ["084601"], "vr": "TM"}, "00080050": {"Value": ["15060734050"], "vr": "SH"}, "00080060": {"Value": ["MR"], "vr": "CS"}, "00080070": {"Value": ["UNKNOWN"], "vr": "LO"}, "00080080": {"vr": "LO"}, "00080081": {"vr": "ST"}, "00080090": {"vr": "PN"}, "00081010": {"vr": "SH"}, "00081030": {"Value": ["PREPROCESS"], "vr": "LO"}, "0008103E": {"Value": ["PREPROCESS"], "vr": "LO"}, "00081040": {"vr": "LO"}, "00081048": {"vr": "PN"}, "00081050": {"vr": "PN"}, "00081060": {"vr": "PN"}, "00081070": {"vr": "PN"}, "00081090": {"Value": ["UNKNOWN"], "vr": "LO"}, "00081110": {"Value": [{"00081150": {"Value": ["1.2.840.10<IP_ADDRESS>.3.1"], "vr": "UI"}, "00081155": {"Value": ["<IP_ADDRESS>5<IP_ADDRESS>.57.20150312.142717.6605503"], "vr": "UI"}}], "vr": "SQ"}, "00081111": {"Value": [{"00081150": {"Value": ["1.2.840.10<IP_ADDRESS>.3.3"], "vr": "UI"}, "00081155": {"Value": ["1.2.840.113619.6.322.240598304242724185143032673684777965063"], "vr": "UI"}}], "vr": "SQ"}, "00081120": {"Value": [{"00081150": {"Value": ["1.2.840.10<IP_ADDRESS>.1.1"], "vr": "UI"}, "00081155": {"Value": ["<IP_ADDRESS>5<IP_ADDRESS>.59.20100828.175520.3010744"], "vr": "UI"}}], "vr": "SQ"}, "00100010": {"Value": [{"Alphabetic": "ANNON"}], "vr": "PN"}, "00100020": {"Value": ["ANNON"], "vr": "LO"}, "00100021": {"vr": "LO"}, "00100030": {"Value": ["20230101"], "vr": "DA"}, "00100032": {"vr": "TM"}, "00100040": {"vr": "CS"}, "00101010": {"vr": "AS"}, "00101030": {"vr": "DS"}, "00102110": {"vr": "LO"}, "00102160": {"vr": "SH"}, "00102180": {"vr": "SH"}, "001021B0": {"vr": "LT"}, "00104000": {"vr": "LT"}, "00120062": {"vr": "CS"}, "00180010": {"Value": ["15    g"], "vr": "LO"}, "00180020": {"Value": ["SE"], "vr": "CS"}, "00180021": {"Value": ["SK"], "vr": "CS"}, "00180022": {"vr": "CS"}, "00180023": {"Value": ["3D"], "vr": "CS"}, "00180025": {"Value": ["N"], "vr": "CS"}, "00180050": {"Value": [0.48828125], "vr": "DS"}, "00180080": {"Value": [600.0], "vr": "DS"}, "00180081": {"Value": [12.319], "vr": "DS"}, "00180083": {"Value": [1.0], "vr": "DS"}, "00180084": {"Value": [127.765418], "vr": "DS"}, "00180085": {"Value": ["1H"], "vr": "SH"}, "00180086": {"Value": [1], "vr": "IS"}, "00180087": {"Value": [3.0], "vr": "DS"}, "00180091": {"Value": [25], "vr": "IS"}, "00180093": {"Value": [100.0], "vr": "DS"}, "00180094": {"Value": [90.0], "vr": "DS"}, "00180095": {"Value": [244.141], "vr": "DS"}, "00181000": {"vr": "LO"}, "00181020": {"Value": ["3.0.0"], "vr": "LO"}, "00181030": {"vr": "LO"}, "00181040": {"Value": ["IV"], "vr": "LO"}, "00181088": {"Value": [0], "vr": "IS"}, "00181090": {"Value": [0], "vr": "IS"}, "00181094": {"Value": [0], "vr": "IS"}, "00181100": {"Value": [250.0], "vr": "DS"}, "00181250": {"vr": "SH"}, "00181310": {"Value": [0, 256, 256, 0], "vr": "US"}, "00181314": {"Value": [90.0], "vr": "DS"}, "00181315": {"Value": ["N"], "vr": "CS"}, "00181316": {"Value": [1.57316], "vr": "DS"}, "00185100": {"Value": ["HFS"], "vr": "CS"}, "0020000D": {"Value": ["2.25.255069905384861154924841019137563439343"], "vr": "UI"}, "0020000E": {"Value": ["2.25.205287721952789789408662186088058372087"], "vr": "UI"}, "00200010": {"vr": "SH"}, "00200011": {"Value": [350], "vr": "IS"}, "00200012": {"Value": [1], "vr": "IS"}, "00200013": {"Value": [1], "vr": "IS"}, "00200032": {"Value": [-92.620185852051, -157.00784301758, -132.19627380371], "vr": "DS"}, "00200037": {"Value": [1.0, 0.0, 0.0, 0.0, 1.0, 0.0], "vr": "DS"}, "00200052": {"Value": ["2.25.115231125129161360422423700315339076734"], "vr": "UI"}, "00200060": {"vr": "CS"}, "00201002": {"Value": [320], "vr": "IS"}, "00201040": {"vr": "LO"}, "00201041": {"Value": [-132.19627], "vr": "DS"}, "00280002": {"Value": [1], "vr": "US"}, "00280004": {"Value": ["MONOCHROME2"], "vr": "CS"}, "00280010": {"Value": [512], "vr": "US"}, "00280011": {"Value": [192], "vr": "US"}, "00280030": {"Value": [1.0, 2.0], "vr": "DS"}, "00280100": {"Value": [16], "vr": "US"}, "00280101": {"Value": [16], "vr": "US"}, "00280102": {"Value": [15], "vr": "US"}, "00280103": {"Value": [1], "vr": "US"}, "00280106": {"Value": [0], "vr": "SS"}, "00280107": {"Value": [739], "vr": "SS"}, "00281050": {"Value": [148.0], "vr": "DS"}, "00281051": {"Value": [11.0], "vr": "DS"}, "00281052": {"Value": [0.0], "vr": "DS"}, "00281053": {"Value": [1.0], "vr": "DS"}, "00281054": {"Value": ["US"], "vr": "LO"}, "00282110": {"Value": ["00"], "vr": "CS"}, "00321033": {"vr": "LO"}, "00380500": {"vr": "LO"}, "00400244": {"Value": ["20150516"], "vr": "DA"}, "00400245": {"Value": ["082225"], "vr": "TM"}, "00400253": {"Value": ["7246.1431760945"], "vr": "SH"}, "00400254": {"vr": "LO"}}'
    resZ = 0.48828125

    for i in range(512):
        ds = pydicom.dataset.Dataset.from_json(header)
        ds[0x0020, 0x0032].value[2] = ds[0x0020, 0x0032].value[2] + i * resZ
        ds.SOPInstanceUID = pydicom.uid.generate_uid(prefix=None)

        arr = np.zeros([512, 192]).astype(np.int16)
        arr[100:400, 30:150] = 150
        if i < 256:
            arr[50:200, 60:90] = 180
        else :
            arr[50:200, 60:90] = 130
        ds.PixelData = arr.tobytes()
        ds.is_little_endian = True
        ds.is_implicit_VR = True

        if isotrop:
            ds.PixelSpacing = [1, 1]
        ds.save_as(os.path.join(dicom_dir, f'MR{i:05}.dcm'))

#####################################
# Create the input dicom image
#####################################
create_input_dicom(dcm_dir, isotrop)

#####################################
# Read the original dicom image
#####################################
reader = sitk.ImageSeriesReader()
dcm_files = reader.GetGDCMSeriesFileNames(dcm_dir)
reader.SetFileNames(dcm_files)
image = reader.Execute()

#####################################
# Create dicom SEG object
#####################################
segmentation_data = process_image(sitk.GetArrayFromImage(image))
segmentation = sitk.GetImageFromArray(segmentation_data)
segmentation.CopyInformation(image)

# Write the template to file
template_json = '{"ContentCreatorName": "Reader1", "ClinicalTrialSeriesID": "Session1", "ClinicalTrialTimePointID": "1", "SeriesDescription": "Segmentation", "SeriesNumber": "300", "InstanceNumber": "1", "BodyPartExamined": "", "segmentAttributes": [[{"labelID": 1,    "SegmentDescription": "test",    "SegmentAlgorithmType": "SEMIAUTOMATIC",    "SegmentAlgorithmName": "Test",    "SegmentedPropertyCategoryCodeSequence": {"CodeValue": "123037004",     "CodingSchemeDesignator": "SCT",     "CodeMeaning": "Anatomical Structure"},    "SegmentedPropertyTypeCodeSequence": {"CodeValue": "91750005",     "CodingSchemeDesignator": "SCT",     "CodeMeaning": "1st Diagonal Coronary Artery"}}]], "ContentLabel": "SEGMENTATION", "ContentDescription": "Image segmentation", "ClinicalTrialCoordinatingCenterName": "dcmqi"}'
with open(template_filename, "w") as write_file:
    json.dump(json.loads(template_json), write_file)

template = pydicom_seg.template.from_dcmqi_metainfo(template_filename)
writer = pydicom_seg.MultiClassWriter(
    template=template,
    skip_empty_slices=False,  # Don't encode slices with only zeros
    skip_missing_segment=False,  # If a segment definition is missing in the
                                 # template, then raise an error instead of
                                 # skipping it.
)

dcm_files = glob.glob(dcm_dir + '\\*')
source_images = [pydicom.dcmread(x, stop_before_pixels=True, force=True) for x in dcm_files]
dcm = writer.write(segmentation, source_images)
dcm.save_as(seg_filename)

#####################################
# Compute the RT struct
#####################################
# Swap axis from Z, Y, X to X, Y, Z
segmentation_data = np.swapaxes(segmentation_data, 0, 2)
segmentation_data = np.swapaxes(segmentation_data, 0, 1)

roi_mask = np.where(segmentation_data == 1, True, False)
rtstruct = RTStructBuilder.create_new(dicom_series_path=dcm_dir)
rtstruct.add_roi(mask=roi_mask, name='seg test', color='cc0000')
rtstruct.save(rt_filename)

When I set isotrop=True -> the x, y resolution is the same and the RT struct creation is fine.
with MITK:

When I set isotrop=False -> the x, y resolution is not the same and the RT is not at the correct place
with MITK:

Changing the line 154 and 184 of the image_helper to :
column_spacing, row_spacing = first_slice.PixelSpacing
fixed my problem

Hello @VincentMorard,
Thanks for opening this issue and  for the detailed description.
I think your issue could be fixed if the 'synthetic' DICOM that you create would store correctly the PixelSpacing of the swapped axes. That should remove the need to have to modify the source-code within the image_helper module, wouldn't it?
Please let me know if doing something like that would also work.
Regarding the force=True of pydicom:
On one hand, I think it is safer have it as False in the tool, as we ensure dicom files are compliant and therefore we know what to expect within the tool, maximizing the chance of it working as intended.
On the other hand, I acknowledge that often in research we had to do some hacks to get newer stuff to work. So I also understand the need for more flexibility.
This is what I propose: I'll work on a way to implement an optional flag to allow the user to import non-compliant dicom files, with the warning that the tool might not work as intended if used like that. Then I'll bring that into the next release.
If you would like to work on that solution and create a PR, that would be fantastic as well and greatly appreciated!
Thank you for using our tool,
Pedro


Preferred way of installation?

Hi,
Have been playing with Reactduino a bit and so far I quite like it. I've got just one question: what is the preferred way to install the library? If I just try to install it with Arduino IDE (or even by copying it to place), my sketches won't find the includes because they're under src. If I copy the source files up to the root folder, Arduino IDE is able to build the project.

I use PlatformIO myself, so I'm not sure on the best approach for the Arduino IDE.
It looks like you can install a ZIP library, which should work with this download link: https://github.com/Reactduino/Reactduino/archive/master.zip which you can get from the main repository.
Can you let me know if that works for you?

That's the way I've done it, but the Arduino IDE doesn't seem to find the headers under the src directory.
Hmm, it seems the Arduino IDE would expect a library.properties file which could then define the header files. I can experiment with it and make a PR if that'd fix it.

That would be very much appreciated!
On 10 Mar 2018 09:11, "Matti Airas"<EMAIL_ADDRESS>wrote:

That's the way I've done it, but the Arduino IDE doesn't seem to find the
headers under the src directory.
Hmm, it seems the Arduino IDE would expect a library.properties file
which could then define the header files. I can experiment with it and make
a PR if that'd fix it.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Reactduino/Reactduino/issues/2#issuecomment-372015590,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGL7M90GKMcURw-t-adP5i56pCYcdYPEks5tc5i_gaJpZM4SjLf0
.

Closed by #4


columns representation

What's up?
Not the top priority of the project, but would it make sense represent columns here:
https://github.com/PRQL/prql/blob/79e5529413fecc66b824d1936e89ac3889f248fb/prql-compiler/src/ast/rq/mod.rs#L77
...with an IndexMap<Cid, RelationColumn>, rather than a Vec of a 2-tuple? That would make the RQ representation more concise.
(It can't be a HashMap because it needs to be ordered)

Yea, I wanted to change that to
    ...
    pub columns: Vec<ColumnDef>
}

struct ColumnDef {
    column: RelationColumn,
    cid: CId,
}

... which is not really more concise.
I treat RQ as a public interface, where third-party library constructs are discouraged.

I treat RQ as a public interface, where third-party library constructs are discouraged.

Yes, good point.
I was predominantly thinking that for the serialization of RQ, it's more concise to have a map — basically half the number of lines — and maps are ordered in JSON / YAML
But I agree for the rust representation it's worse to require a 3rd part library.
Would you support having it as a map the Serde implementation? (super low priority, maybe not worth the discussion)

and maps are ordered in JSON / YAML

Not really. Many implementations may treat them that way, but the standard says:

An object is an unordered set of name/value pairs.

Not really. Many implementations may treat them that way, but the standard says:

An object is an unordered set of name/value pairs.

Ah, good find.
OK then, I guess this isn't going to work. Thanks for engaging...


added org.apache.bcel.classfile.FieldOrMethod.getDeclaringClassName()

This pull request adds getDeclaringClassName() in FieldOrMethod. It can be helpful to know in which class a field or method is declared.

Codecov Report

Merging #185 (704d62c) into master (50afad5) will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##             master     #185      +/-   ##
============================================
+ Coverage     61.65%   61.66%   +0.01%
- Complexity     3655     3657       +2
============================================
  Files           363      363
  Lines         15633    15638       +5
  Branches       1950     1950
============================================
+ Hits           9638     9643       +5
  Misses         5121     5121
  Partials        874      874

Impacted Files
Coverage Δ

...in/java/org/apache/bcel/classfile/ClassParser.java
59.77% <100.00%> (+0.94%)
:arrow_up:

.../java/org/apache/bcel/classfile/FieldOrMethod.java
54.68% <100.00%> (+2.22%)
:arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

Hi @garydgregory
Please help moving forward with this PR
Thanks !

This PR has no description, why is it needed?

not 100% needed, I can propagate the declaring class name instead

Following the yagni paradigm. I already have a satisfying workaround for this.


Setup script for more collaborators

Hi!
Setting up a cloned repo to create PRs is quite cumbersome, especially due to the native skia submodule.
Getting that to build on a windows machine is quite a hassle.
It would be so nice if there was an (optional) setup script installing all the stuff required to actually build skia and also building it.
Or at least have a good documentation. (I know it is not on you to document how to build skia, but every time I have I struggle quite a bit to eventually get it to compile...)
So this is just a "hope it will be implemented some time" "issue" - kind of a reminder ;-)

I will try and do this. But,  I am wondering how you are building on Windows? I just run:
./bootstrapper.ps1 -Target externals-windows

And that does everything. (Make sure the git submodule update --init --recursive has run)
I dod that from a fresh Windows install with VS2015.

well... not so much:

PS D:\Code\SkiaSharp_gd> git submodule update --init --recursive
PS D:\Code\SkiaSharp_gd> ./bootstrapper.ps1 -Target externals-windows
Feeds used:
  C:\Users\alexa\AppData\Local\NuGet\Cache
  C:\Users\alexa\.nuget\packages\
  https://www.nuget.org/api/v2

Alle in "D:\Code\SkiaSharp_gd\tools\packages.config" aufgeführten Pakete sind bereits installiert.
Analyzing build script...
Processing build script...
Installing addins...
Compiling build script...
Error: D:/Code/SkiaSharp_gd/cake/UtilsManaged.cake(4,27): error CS0103: The name 'DotNetCoreRestoreVerbosity' does not e
xist in the current context
PS D:\Code\SkiaSharp_gd>

???
Maybe your cake folder is out of date. Just delete everything in your /tools folder except the mdoc folder, .gitignore and packages.config.

You will also need .netcore installed

you were absolutely right: Now it worked up to this point

PS D:\Code\SkiaSharp_gd> ./bootstrapper.ps1 -Target externals-windows
Feeds used:
  C:\Users\alexa\.nuget\packages\
  https://www.nuget.org/api/v2

Restoring NuGet package NUnit.ConsoleRunner.3.6.0.
Restoring NuGet package Microsoft.DotNet.BuildTools.GenAPI.1.0.0-beta-00081.
Restoring NuGet package Cake.0.17.0.
Adding package 'NUnit.ConsoleRunner.3.6.0' to folder 'D:\Code\SkiaSharp_gd\tools'
Adding package 'Cake.0.17.0' to folder 'D:\Code\SkiaSharp_gd\tools'
Adding package 'Microsoft.DotNet.BuildTools.GenAPI.1.0.0-beta-00081' to folder 'D:\Code\SkiaSharp_gd\tools'
Added package 'NUnit.ConsoleRunner.3.6.0' to folder 'D:\Code\SkiaSharp_gd\tools'
Added package 'Microsoft.DotNet.BuildTools.GenAPI.1.0.0-beta-00081' to folder 'D:\Code\SkiaSharp_gd\tools'
Added package 'Cake.0.17.0' to folder 'D:\Code\SkiaSharp_gd\tools'
Analyzing build script...
Processing build script...
Installing addins...
Downloading and installing Roslyn...
Installing packages (using https://packages.nuget.org/api/v2)...
Copying files...
Copying Roslyn.Compilers.CSharp.dll...
Copying Roslyn.Compilers.dll...
Deleting installation directory...
Compiling build script...
 Nov 29 17:05:14 2016"
        Key: USERDOMAIN_ROAMINGPROFILE  Value: "LAPTOP-3M6KT9KO"
        Key: PUBLIC     Value: "C:\Users\Public"
Error: Could not reach target 'externals-windows' since it was skipped due to a criteria.

maybe the bootstrapper should  by default delete all stuf of the tools folder, just to be sure?
also for ".netcore": take a look at Automappers psake build script
There it automagically installs .netcore ;-)  (function Install-Dotnet)

don't get me wrong: I know it is a lot to ask to keep these installs so smooth - especially with such a complicated repository.
but your PR count and community acceptance would greatly benefit of this :)

the externals-windows was skipped due to the fact that all the output files are there. you will also have to delete the /native-builds/libs/windows folder
Or, what I do for dev, just comment out the checks: https://github.com/mono/SkiaSharp/blob/master/cake/BuildExternals.cake#L157-L159
I suppose I have been doing this so long I just know what to do :) Also, the script is really aimed for the managed dev - I try not to build the native bits too often.
And, for the average joe who wants to build from scratch with an old checkout:
.\bootstrapper.ps1 -Target clean

That will nuke all!

Any and all feedback is welcomed as I need to get this easy for anyone.

in that case you only have to change your readme to
"
install .netcore (with a link to the installer, or even a script?)
run
.\bootstrapper.ps1 -Target clean
then .\bootstrapper.ps1 -Target externals-windows
"
that should do it for now

Good point!
Is your build building?

seems like it.
As for "installing dotnet automatigally", the following code actually works (I just copied it from the Automapper build script I referenced earlier)

function Install-Dotnet
{
    $dotnetcli = where-is('dotnet')

    if($dotnetcli -eq $null)
    {
		$dotnetPath = "$pwd\.dotnet"
		$dotnetCliVersion = if ($env:DOTNET_CLI_VERSION -eq $null) { 'Latest' } else { $env:DOTNET_CLI_VERSION }
		$dotnetInstallScriptUrl = 'https://raw.githubusercontent.com/dotnet/cli/rel/1.0.0/scripts/obtain/install.ps1'
		$dotnetInstallScriptPath = '.\scripts\obtain\install.ps1'

		md -Force ".\scripts\obtain\" | Out-Null
		curl $dotnetInstallScriptUrl -OutFile $dotnetInstallScriptPath
		& .\scripts\obtain\install.ps1 -Channel "preview" -version $dotnetCliVersion -InstallDir $dotnetPath -NoPath
		$env:Path = "$dotnetPath;$env:Path"
	}
}

function where-is($command) {
    (ls env:\path).Value.split(';') | `
        where { $_ } | `
        %{ [System.Environment]::ExpandEnvironmentVariables($_) } | `
        where { test-path $_ } |`
        %{ ls "$_\*" -include *.bat,*.exe,*cmd } | `
        %{  $file = $_.Name; `
            if($file -and ($file -eq $command -or `
			   $file -eq ($command + '.exe') -or  `
			   $file -eq ($command + '.bat') -or  `
			   $file -eq ($command + '.cmd'))) `
            { `
                $_.FullName `
            } `
        } | `
        select -unique
}

now if you included those functions in "bootstrapper.ps1", you could ensure that dotnet was installed by simply calling
install-dotnet
I think that's pretty awesome 👍

oh crap... wait - I just noticed that the install script it is supposed to download does not exist anymore.
 Thank you so much! 🥇

Although not perfect yet, I am closing this as I think many issues have been resolved by switching to the new GN build system.
This should be all that is needed: https://github.com/mono/SkiaSharp#building-skiasharp
I decided not to install anything during the build as this may cause issues on existing machines.


Fix executing empty sql statement in preactions and postactions

Say for example:
you need to execute preactions/postactions
val preactions = "update table1 set col1=2; update table2 set col2 = 2; "
Looks nothing wrong, correct?
Only when you try to execute it, you will get weird error:
Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: 0
This is because of Line 248 and Line 260 in Parameters.scala
def preActions: Array[String] = parameters("preactions").split(";")
def postActions: Array[String] = parameters("postactions").split(";")
the above functions are called in RedshiftWriter.scala LN: 132 and LN: 195
parameters("preactions").split(";")
-- this returns an array of 3 items, with the 3rd item being empty string (only spaces) - see trailing spaces in
val preactions = "update table1 set col1=2; update table2 set col2 = 2; "
and then it tries to execute sql query which is essentially just "spaces"
The fix is to just trim preactions and postactions so that the trailing spaces are not considered as query to be executed

I am working on the fix and will soon raise a PR

https://github.com/spark-redshift-community/spark-redshift/pull/57

Also, when you get to this, could you please update the README and CHANGELOG accordingly? (A patch bump is fine since this is a backward compatible bugfix)


Simple API endpoint to query latest registered pets

For now this could be very simple and read-only I believe.
But as I would like it to not be a problem for resource usage in case of usage abuse, I think we need something to control the rate of requests.

Would setting a cache be an acceptable solution, in the use case that you mention?
As I believe the primary resource usage would be the database query, eliminating this to a reasonable invalidation period, would help in reducing resource usage?
What do you think?
Reference : https://www.dabapps.com/blog/api-performance-profiling-django-rest-framework/

@rohanthacker sure!
I was thinking about this too.
There's a few "discussion" about this too in #17.
But I'd like to leave the cache layer modification to be made before everything, if and when we really come to need it.
Right now what I would really like is to add an action in the admin to share the selected pets in the Facebook page. The sharing stuff is already a Django command.
Would you like to do/discuss this?
(I'll create a new issue)


Use latest attainment date as Sisu CSV date

The option of using the latest attainment date as the date in the Sisu CSV came up. Should this be done or should the teacher still need to set it manually?

The teacher should not have to set a completion date when exporting a Sisu CSV. The date is automatically determined by the latest completed attainment grade for a given student.

'If not provided, the default will be course instance ending date.'
Should this text be changed to
'If not provided, for each student their latest attainment date will be used.'


BAEL-7636 Add the missing plugin versions in the tutorial repo

-Added missing plugin versions

Hey @lor6 ,
Can this https://stackoverflow.com/questions/4856307/when-maven-says-resolution-will-not-be-reattempted-until-the-update-interval-of be related?

@dkapil  I tried to restart the build after 24h. Still failing :(

@lor6 so spring-swagger-codegen-app module can not find spring-swagger-codegen-client module.
I added an extra line in readme.md of spring-swagger-codegen-client so that its build can be triggered in travis and it may get available for spring-swagger-codegen-app. Let's see...

@lor6 It worked. :+1:


NPM package

Stumbled upon this library - and it looks awesome.
If you create a release (see #11 ) - would it be possible to submit this to npm?
I can wrap it in a package myself, but it'd be awesome if you'd add the required bits (a suitable package.json, mostly?) for npm to be happy. Would you accept a PR for that?

Do you want to use this lib in node? Or do you just want to use npm as a package manager for browser applications? First will not work, caused of the different handling of ajax calls. 2nd is planed but will take some time. Any PR is welcome if you want to contribute.

The latter. I'd use this library in the browser, but use webpack and npm to manage the build/dependencies etc. I'll see if I can help with the package.

npm package is registered with the new version.
npm install o.js


import-export not reading column from .csv unless I save it in TextEdit first.

I am using django-import-export to upload a csv file. I cannot get it to read a certain column on upload in the admin or in my view unless I save it in Mac's TextEdit app. The rows are straight forward with no unusual characters.
An example row is:
ASIN,Title,P-index,Category,Brand,Fulfillment,"Size Tier","Number of Images","Variation Count",Weight,BSR,Price,"Monthly Sales","Net Price","Monthly Revenue","Review Count","Reviews Rating",Sellers,"Last Year Sales","Sales Year Over Year","Sales Trend (90 days)","Price Trend (90 days)","Best Sales Period","Sales to Reviews" B00167YCSM,"HERSHEY'S Milk Chocolate Bulk Holiday Candy, 4.4 Ounce, Extra Large Bars, 12 Count",,"Grocery & Gourmet Food",HERSHEY'S,Amazon,"Large Standard-Size",6,4,3.9903669422,747,17.76,17055,8.92,294710.4,2861,4.8,6,,,2471,-31,"Dec, 2020",6.23
It is the ASIN column with B00167YCSM as the value (or any other similar ASIN value). Mac users are able to upload (with ASIN row showing) only if they open the csv in TextEdit and save it, but Windows users are not able to upload csv files no matter what they use to save the file with.
Any ideas on what could be causing this?

Please dont create more than one issue for the same problem. Closing dupe of #1243


Is there a way to obtain the confidence of each landmark?

Yes, it is easy to add a subnet in the head to do the classification. And it will be more convenient to do tracking with the face landmark model.
However, i got no data for that .


Global dependencies installed with Node from nvm are not working

Operating system and version: CentOS Linux release 7.6.1810 (Core)
nvm debug output:

nvm --version: v0.33.11
$SHELL: /bin/bash
$SHLVL: 1
$HOME: /root
$NVM_DIR: '$HOME/.nvm'
$PATH: $NVM_DIR/versions/node/v18.12.0/bin:/opt/rh/devtoolset-8$HOME/usr/bin:/data:/usr/local/maven/apache-maven-3.6.3/bin:/usr/software/jdk-11.0.9/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/software/gradle-4.6/bin:/usr/local/bin:/data/android/sdk/cmdline-tools/latest/bin:/data/android/sdk/tools/bin:/data/android/sdk/ndk/25.0.8775105:$HOME/bin
$PREFIX: ''
$NPM_CONFIG_PREFIX: ''
$NVM_NODEJS_ORG_MIRROR: ''
$NVM_IOJS_ORG_MIRROR: ''
shell version: 'GNU bash， 版本 4.2.46(2)-release (x86_64-redhat-linux-gnu)'
uname -a: 'Linux 3.10.0-1160.53.1.el7.x86_64 #1 SMP Fri Jan 14 13:59:45 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux'
OS version: CentOS Linux 7 (Core)
curl: curl 是 /usr/bin/curl, curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.53.1 zlib/1.2.7 libidn/1.28 libssh2/1.8.0
wget: wget 是 /usr/bin/wget, GNU Wget 1.14 在 linux-gnu 上编译。
git: git 是 /usr/bin/git, git version <IP_ADDRESS>
grep: grep 是 `grep --color=auto' 的别名, grep (GNU grep) 2.20
awk: awk 是 /usr/bin/awk, GNU Awk 4.0.2
sed: sed 是 /usr/bin/sed, sed (GNU sed) 4.2.2
cut: cut 是 /usr/bin/cut, cut (GNU coreutils) 8.22
basename: basename 是 /usr/bin/basename, basename (GNU coreutils) 8.22
rm: rm 是 `rm -i' 的别名, rm (GNU coreutils) 8.22
mkdir: mkdir 是 /usr/bin/mkdir, mkdir (GNU coreutils) 8.22
xargs: xargs 是 /usr/bin/xargs, xargs (GNU findutils) 4.5.11
nvm current: v18.12.0
which node: $NVM_DIR/versions/node/v18.12.0/bin/node
which iojs: which: no iojs in ($NVM_DIR/versions/node/v18.12.0/bin:/opt/rh/devtoolset-8$HOME/usr/bin:/data:/usr/local/maven/apache-maven-3.6.3/bin:/usr/software/jdk-11.0.9/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/software/gradle-4.6/bin:/usr/local/bin:/data/android/sdk/cmdline-tools/latest/bin:/data/android/sdk/tools/bin:/data/android/sdk/ndk/25.0.8775105:$HOME/bin)
which npm: $NVM_DIR/versions/node/v18.12.0/bin/npm
npm config get prefix: $NVM_DIR/versions/node/v18.12.0
npm root -g: $NVM_DIR/versions/node/v18.12.0/lib/node_modules

nvm ls output:

        v8.11.4
->     v18.12.0
default -> 18.12.0 (-> v18.12.0)
node -> stable (-> v18.12.0) (default)
stable -> 18.12 (-> v18.12.0) (default)
iojs -> N/A (default)
lts/* -> lts/iron (-> N/A)
lts/argon -> v4.9.1 (-> N/A)
lts/boron -> v6.17.1 (-> N/A)
lts/carbon -> v8.17.0 (-> N/A)
lts/dubnium -> v10.24.1 (-> N/A)
lts/erbium -> v12.22.12 (-> N/A)
lts/fermium -> v14.21.3 (-> N/A)
lts/gallium -> v16.20.2 (-> N/A)
lts/hydrogen -> v18.20.4 (-> N/A)
lts/iron -> v20.15.1 (-> N/A)

How did you install nvm?

install script in readme
What steps did you perform?
I installed nvm following the instructions in the official documentation.
I used nvm to install Node.js and then ran the command npm install -g webpack.
What happened?
After running npm install -g webpack, the webpack executable was not found in the bin folder.
What did you expect to happen?
I expected the webpack executable to be placed in the bin directory for global executables.
Is there anything in any of your profile files that modifies the PATH?

.bash_profile

If you are having installation issues, or getting "N/A", what does curl -I --compressed -v https://nodejs.org/dist/ print out?

* About to connect() to nodejs.org port 443 (#0)
*   Trying <IP_ADDRESS>...
* Connected to nodejs.org (<IP_ADDRESS>) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=*.nodejs.org
*       start date: 2月 28 00:00:00 2024 GMT
*       expire date: 3月 30 23:59:59 2025 GMT
*       common name: *.nodejs.org
*       issuer: CN=Sectigo RSA Domain Validation Secure Server CA,O=Sectigo Limited,L=Salford,ST=Greater Manchester,C=GB
> HEAD /dist/ HTTP/1.1
> User-Agent: curl/7.29.0
> Host: nodejs.org
> Accept: */*
> Accept-Encoding: deflate, gzip
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Thu, 07 Nov 2024 03:43:35 GMT
Date: Thu, 07 Nov 2024 03:43:35 GMT
< Content-Type: text/html
Content-Type: text/html
< Connection: keep-alive
Connection: keep-alive
< Cache-Control: public, max-age=3600, s-maxage=14400
Cache-Control: public, max-age=3600, s-maxage=14400
< Last-Modified: Thu, 07 Nov 2024 02:05:22 GMT
Last-Modified: Thu, 07 Nov 2024 02:05:22 GMT
< CF-Cache-Status: HIT
CF-Cache-Status: HIT
< Age: 5886
Age: 5886
< Vary: Accept-Encoding
Vary: Accept-Encoding
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< Server: cloudflare
Server: cloudflare
< CF-RAY: 8dea51e6ec0b822a-IAD
CF-RAY: 8dea51e6ec0b822a-IAD
< Content-Encoding: gzip
Content-Encoding: gzip

<
* Connection #0 to host nodejs.org left intact

what do npm ls -g --depth=0 and which webpack print out?

[root@ecs-213894-0003 inz_frontend_production]# npm ls -g --depth=0
/root/.nvm/versions/node/v18.12.0/lib
├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>└──<EMAIL_ADDRESS>

I ran npm install in my Vue project, but the corresponding executable files were not generated in the node_modules/.bin directory. I'm not sure if this is `related.```

(either way, yes, this looks to be npm-related, not nvm-related, since nvm seems to have set npm and node up properly, but I'm happy to keep debugging)

ls -la /root/.nvm/versions/node/v18.12.0/bin
drwxr-xr-x 2 1001 1001     4096 10月 25 2022 .
drwxr-xr-x 6 root root     4096 7月  11 10:19 ..
lrwxrwxrwx 1 1001 1001       45 10月 25 2022 corepack -> ../lib/node_modules/corepack/dist/corepack.js
-rwxr-xr-x 1 1001 1001 86795368 10月 25 2022 node
lrwxrwxrwx 1 1001 1001       38 10月 25 2022 npm -> ../lib/node_modules/npm/bin/npm-cli.js
lrwxrwxrwx 1 1001 1001       38 10月 25 2022 npx -> ../lib/node_modules/npm/bin/npx-cli.js

what if you npx npm@latest install -g webpack? It’s possible a later version of npm fixes it.

The method is not quite working either. I would like to know how the executable files in versions/node/v18.12.0/bin are generated. Is it a modification of npm's configuration, or is it implemented by nvm itself?

Thank you for helping me investigate this issue. I’ve been debugging with the npm source code on the server, but I still can't create the .bin directory. I’ll continue looking into it, but I’m sure it's unrelated to nvm.

I found the reason. It's because the server's .npmrc file had bin-links=false configured.

aha, that would do it :-) glad you figured it out!


basic connector widget added

jupysql-interactive-connection implemented as JupyterLab widget. - ConnectorWidget

Allows to connect to supported DBs using UI.
Allows to create a new connection.
The active connection button is marked with the primary color.
Once a connection is created, the user automatically connects to it.
Notifies the user if the connection name already exists (connection name is the alias)
connections.ini file added and stores all information
Keyring removed for simplicity
Delete connection added with double validation
Databases added: DuckDB, SQLite, PostgreSQL, MySQL, MariaDB, and SnowFlake.
Displays errors

example:

Original issue: #16

please update playwright snapshots

@edublancas Thanks for the feedback! agree with everything.
Addressing:

Default connections - delete / load
Connection name - alias
Delete connection instructions
Display error
Code cleanup

How do you think we should use telemetry here? via jupysql library or from the plugin?

let's put the telemetry code in jupysql.

please address the merging conflicts. once that's done, looks like this will be ready to merge, right?

please address the merging conflicts. once that's done, looks like this will be ready to merge, right?

Yes. Please see if the design/logic/functionality is OK.


C++ pattern code added by Fatimareefat

Hey!
Can you just tell me what I am doing wrong or something?


[FLINK-6018] Add tests for KryoSerializer restore with registered types

This is the result of the discussion in #3534.
I changed TypeSerializer.isCompatibleWith() to TypeSerializer.canRestoreFrom because the relation is not necessarily symmetric.
I added a KryoSerializer.canRestoreFrom() that only allows restoring when we previously didn't have registered types/serializers.
I added a whole bunch of tests in StateBackendTestBase, this should be review most thoroughly.
R: @StephanEwen and @tzulitai because this probably is interesting with the serialiser update story that you're working on.

+1 to the rename. I was thinking the same here :-)

@aljoscha I've been trying out this PR to also test some Kryo behaviours. I can address the above comments along the way and open a PR against this one :-)

Here's my follow-up fixes for this: https://github.com/tzulitai/flink/tree/FLINK-6018
It also includes a rebase for the merge conflict + test fixes.
We can continue with this after Aljoscha comes back from vacation :)

@tzulitai I would suggest you open a PR with two commits: mine and your changes on top. What do you think? Then I would close this PR.

@aljoscha sure, I'll do that.

Done. Re-opened at #3603.

Closing in favour of #3603


Confirm all env variables on startup

If I'm going to fail at Part 3, I've still generated an image in Part 2, and paid for it and everything, but nobody will get to see that bear.
Make some kind of startup function that confirms all required variables are present before doing ANYTHING.

Added with  #73ae94d


Add support for MS Teams web

closes #11
It's fully working, tested on exam today.

@Lumm1t can you review this PR?

Sure, can you change commit name to smth like "feat(manifest): support for ms teams"?
https://gist.github.com/joshbuchea/6f47e86d2510bce28f8e7f42ae84c716

@Lumm1t done


add template_card message support for WeCom notification provider

⚠⚠⚠ Since we do not accept all types of pull requests and do not want to waste your time. Please be sure that you have read pull request rules:
https://github.com/louislam/uptime-kuma/blob/master/CONTRIBUTING.md#can-i-create-a-pull-request-for-uptime-kuma
Tick the checkbox if you understand [x]:

[x] I have read and understand the pull request rules.

Description
add markdown message support for WeCom notification provider
Type of change

New feature (non-breaking change which adds functionality)

Checklist

[x] My code follows the style guidelines of this project
[x] I ran ESLint and other linters for modified files
[x] I have performed a self-review of my own code and tested it
[x] I have commented my code, particularly in hard-to-understand areas (including JSDoc for methods)
[x] My changes generates no new warnings
[ ] My code needed automated testing. I have added them (this is optional task)

Screenshots (if any)

Event
before
after

UP

DOWN

Cert-expiry

Testing

This msgtype is not one which is officially recognised by the api as far as I see it. Am I looking at the wrong docs? (I can't read Chinese => this might be a language barrier)
https://developers.weixin.qq.com/miniprogram/en/dev/api-backend/open-api/customer-message/customerServiceMessage.send.html

Maybe English version is outdated. it's documented in this Chinese one.
WeCom Doc

Agree, much nicer.
From a UX perspective I feel like switching these two fieds might be benefitial.
What do you think?

Another thing that I forgot in the second review (we can not do this ^^).
Do you think the call-to-action buttons as in buildActions from https://github.com/louislam/uptime-kuma/pull/3886 might be possible?

Agree, much nicer. From a UX perspective I feel like switching these two fieds might be benefitial. What do you think?

The reason for this arrangement is that this way a quick status will show up in brief view. Other wise you have to click into that message to see the status. Or we can just discard that most up title.

Another thing that I forgot in the second review (not adding this would also be fine ^^). Do you think the call-to-action buttons as in buildActions from #3886 might be possible?

There's already a builtin call-to-action function, you click the message it will jump to the monitor url. I just did not find a quick way to get the site url.

I think it looks better without the title bar. Push the change.

@CommanderStorm
yes, those buttons could be added as a link, they are called jump_list. The thing is the builtin jump URL is mandatory, in case of no link present, the whole message just wouldn't be sent. The same goes for jump_list. That's my last commit is for. I could add other entries to jump_list, but if there's no URL to be retrieved and it's not properly tested against all edge cases, that's just another failure point. That's the reason I first try to use markdown instead of template_card.

I actually agree with you, but as I said, card_action is mandatory for template_card.

I just went over the discussion to look if there is something left open and two things are currently blocking a merge:

I think this part fell under the table. Is this intentional?

The same goes for jump_list: in case no links exist, don't add the list.
Going by the docs, that should work

there have been some other changes to wechat in the master branch. could you merge these changes into your PR?

@darkclip gentle ping ^^


adding tuple

Adding tuple docs
deleting flatten as a function as well (old)

@paddyjok Please take a look. I tried using tuple in a select query, and it failed, so I think it's only meant to work with BULK INSERT transforms? As such I put it there. If that's not the case, it can be moved around and a bug ticket can be put in.


Fix missing status definition in v1 CRD

As suggested by https://github.com/kubernetes-sigs/aws-iam-authenticator/issues/406#issuecomment-947592660
This fixes the v1 CRD, otherwise aws-iam-authenticator would reporting syncing the CRD but all auth requests were rejected.

/assign @wongma7

/lgtm


prior needs .mean and .stddev for standardize

We are using the PyTorch distribution attributes .mean and .stddev to build the standardize transform. This gives another requirement for the prior:
https://github.com/mackelab/sbi/blob/dac42ed0ebdefff9037ebf4f782c6d1fed9ecfc4/sbi/utils/get_nn_models.py#L44-L46
Instead, we could just sample from the prior, which should be cheap, an estimate mean and std from samples, e.g.,
theta = prior.sample((1000, ))
mean, std = (theta.mean(axis=0), theta.std(axis=0))

Could sampling be the fallback way to compute .mean, .std when not defined? If we are always wrapping, we could actually ship that method in the wrapping distribution (even if named .meanstd_from_samples(axis).


Constraint on types in Bulk POST

Currently, the bulk extension just says that every resource I POST must contain a "type" member, but it doesn't constrain those types at all, or say how the server should respond to bizarre types.
For example, this should pretty obviously not be allowed:
POST /photos
{
  "data": [{"type": "people", "name": "New Person!"}]
}

That implies a basic restriction like "The "type" value provided for each resource must be among types that constitute the collection represented by the URI that you're POSTing to."
But, beyond that basic restriction, I'm wondering if there should be more guidance. E.g., perhaps the server MAY reject the request if all the "type"s don't match? Etc.

I think a 422 (Unprocessable entity) would be appropriate. I'm not sure we need to provide guidance for every single error condition, but this might be a good one to include.

Yup. And, if we do provide guidance here, presumably it should match the guidance we give for the non-bulk equivalent of this error. For that, we currently require a 409:

A server MUST return 409 Conflict when processing a POST request in which the resource's type does not match the server's endpoint.

This seems good @ethanresnick, would you mind opening a PR?

I opened a PR that starts work on this!


Possible App Store release?

Are you planning on bringing this screensaver to the Apple App Store like you did with Irvue? Would be much easier to update etc.

As I know, Mac App Store doesn't support native screensavers (with .saver extension).

Oh you're right. Too bad.


Check RBAC permissions and add docs

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: null
  name: k6-operator-manager-role
rules:
- apiGroups:
  - apps
  resources:
  - deployments
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - batch
  resources:
  - jobs
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - services
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - pods
  - pods/log
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - list
  - get
  - watch

This seems dangerous. Can documentation be provided to show how these permissions may be more restricted?

Just a note; it's quite likely that using operator with --out cloud option and without it can have different lists of RBAC. If so, it makes sense to check all RBAC and make separate lists.

Are there any news yet? I'm looking for a way to deploy the operator + all tests in a single namespace with no access rights outside this namespace.

@realHarter, this issue is more about validation and documentation rather than any specific changes. But it's quite possible to deploy k6-operator and the tests in the same namespace as it is. Have you encountered an issue while trying to do that?

@yorugac I haven't tested alot yet. But so far it's working for me after rewriting all ClusterRoles to Roles, declaring the crds as namespaced and setting the WATCH_NAMESPACE env variable to the single namespace I'm deploying too.

Thanks for details. A guide for such a setup would likely be nice :+1:

@yorugac when you say

But it's quite possible to deploy k6-operator and the tests in the same namespace as it is.

Were you expecting this to be done by changing configuration as @realHarter did or is there a cleaner way with kustomize to do this?  I'm just playing with kustomize for the first time today so I haven't yet determined if this is possible.

@yorugac I haven't tested alot yet. But so far it's working for me after rewriting all ClusterRoles to Roles, declaring the crds as namespaced and setting the WATCH_NAMESPACE env variable to the single namespace I'm deploying too.

Hello @realHarter, can you provide me the manifest file with this changes please ? thank you

I can confirm @realHarter's approach. Well I didn't have to adapt the CRDs because in the latest helm release they already have the scope "Namespaced". The only things that needed to be changed was to change the clusterRole to role (just change Kind from ClusterRole to Role and add the namespace (and also adapt the connected clusterBinding -> same approach):
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: {{ include "k6-operator.fullname" . }}-manager-role
  namespace: {{- include "k6-operator.namespace" . }}
...

Setting the env variable was also necessary. I added that to the deployment.yaml so that it is set automatically for each env the k6-operator is deployed to. Just under "env" on the "manager" container add this:
env:
  - name: WATCH_NAMESPACE
    value: {{- include "k6-operator.namespace" . }}

@FloGro3, thanks for sharing the details here :raised_hands:

the WATCH_NAMESPACE env variable and its function could be documented better. Maybe worth a small section in the readme?

Indeed; we obviously forgot to document it :smile: Added it now, thanks!
By the way, all available options for Helm can be found over here:
https://github.com/grafana/k6-operator/blob/main/charts/k6-operator/README.md
It's an auto-generated documentation for values.yaml file. So you don't need to tinker with the template directly: it can be done via manager.env option.


Project deletion metadata audit log

Summary
Provide a brief overview of the changes and the issue being addressed.
Explain the rationale and any background necessary for understanding the changes.
List dependencies required by this change, if any.
In order for us to be able to calculate customer churn, we need to keep track of the deleted projects/accounts on Minder and the relevant metadata from a business perspective.
Fixes https://app.zenhub.com/workspaces/minder-sprints-666857b34fcef725d2efba7c/issues/gh/stacklok/infra/1906
Change Type
Mark the type of change your PR introduces:

[ ] Bug fix (resolves an issue without affecting existing features)
[x] Feature (adds new functionality without breaking changes)
[ ] Breaking change (may impact existing functionalities or require documentation updates)
[ ] Documentation (updates or additions to documentation)
[ ] Refactoring or test improvements (no bug fixes or new functionality)

Testing
Outline how the changes were tested, including steps to reproduce and any relevant configurations.
Attach screenshots if helpful.
Review Checklist:

[x] Reviewed my own code for quality and clarity.
[x] Added comments to complex or tricky code sections.
[ ] Updated any affected documentation.
[ ] Included tests that validate the fix or feature.
[ ] Checked that related changes are merged.

coverage: 53.626% (-0.08%) from 53.703%
when pulling be8e22347bee87c5d8f7de21efad1fd3081f13f1 on project-deletion-audit
into 0cbe5ab270b7d3510677488c81904e9a0942a0c1 on main.


Text drawer can't draw composite instructions

composite gates (i.e. without classical registers) work fine, but with classical registers it cannot be drawn. The logic is here:
https://github.com/Qiskit/qiskit-terra/blob/109d8ea562987b9309501b9260c80180216061eb/qiskit/visualization/text.py#L849
latex and mpl can do this.

Example code?

It this the expected output?
q = QuantumRegister(4)
c = ClassicalRegister(4)
circ1 = QuantumCircuit(q, c, name='my instruction')
circ1.measure(q, c)
inst = circ1.to_instruction()

circ2 = QuantumCircuit(q, c, name='circ2')
circ2.append(inst, q[:], c[:])
circ2.draw(output='...')

Latex

MPL

Text
Error.

No mpl and latex also need to be fixed. The box needs to extend over the classical registers as well. But at least they draw something. I would say text is priority since that's the default drawer.


Building on Humble

I'd like to be able to use OpenDDS with ROS humble, I've put some time to get the library to build on humble (just marking things as unimplemented for now).
I'm getting this error now when trying to use it (launching the topic monitor in rqt), I'm struggling to find some clear documentation to 'creating a rmw' and what needs to be set/enabled or implemented to get something to work.
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-vscode'
(124728|124795) WARNING: MulticastManager::join: failed to join group <IP_ADDRESS>:7152 on veth08d9df8/<IP_ADDRESS> (0x55a2734e55a8 joined count 4): Unknown error -2

>>> [rcutils|error_handling.c:108] rcutils_set_error_state()
This error state is being overwritten:

  'Handle's typesupport identifier (rosidl_typesupport_c) is not supported by this library, at ./src/type_support_dispatch.hpp:113'

with this new error message:

  'type support implementation 'rosidl_typesupport_c' does not match 'rosidl_typesupport_opendds_cpp', at /workspaces/ws_humble/rmw_opendds/rmw_opendds_cpp/include/rmw_opendds_cpp/RmwAllocateFree.hpp:84'

rcutils_reset_error() should be called after error handling to avoid this.
<<<

>>> [rcutils|error_handling.c:108] rcutils_set_error_state()
This error state is being overwritten:

  'type support implementation 'rosidl_typesupport_c' does not match 'rosidl_typesupport_opendds_cpp', at /workspaces/ws_humble/rmw_opendds/rmw_opendds_cpp/include/rmw_opendds_cpp/RmwAllocateFree.hpp:84'

with this new error message:

  'DDSPublisher failed, at /workspaces/ws_humble/rmw_opendds/rmw_opendds_cpp/src/rmw_publisher.cpp:128'

rcutils_reset_error() should be called after error handling to avoid this.
<<<
(124728|124795) WARNING: MulticastManager::join: failed to join group <IP_ADDRESS>:7150 on veth08d9df8/<IP_ADDRESS> (0x55a2734ef448 joined count 4): Unknown error -2
Segmentation fault (core dumped)

Do you have changes in a forked repo?
Does something like https://github.com/OpenDDS/rmw_build/blob/master/run_ci_test.sh work on Humble?

Woops, forgot to link the fork, https://github.com/tmayoff/rmw_opendds/tree/humble. I didn't use that repo to test, but our repo for work. On humble there were a bunch of build errors, changes to the rmw.h header (and others)

There's a lot that needs to be done as prerequisites for getting rmw_build to work in humble
When going through the rmw_build repo I eventually get these build errors, I feel like this might be due to the opendds install I have in the image, being a custom debian repo I'm adding.
root@12e5b606b02d:/opt/workspace# ./rmw_build/build_examples.sh
not found: "/opt/workspace/install/builtin_interfaces/share/builtin_interfaces/local_setup.bash"
not found: "/opt/workspace/install/unique_identifier_msgs/share/unique_identifier_msgs/local_setup.bash"
not found: "/opt/workspace/install/action_msgs/share/action_msgs/local_setup.bash"
Starting >>> builtin_interfaces
Starting >>> unique_identifier_msgs
--- stderr: builtin_interfaces
In file included from /opt/workspace/build/builtin_interfaces/rosidl_generator_cpp/builtin_interfaces/msg/duration.hpp:7,
                 from /opt/workspace/build/builtin_interfaces/rosidl_typesupport_opendds_cpp/builtin_interfaces/msg/duration__rosidl_typesupport_opendds_cpp.hpp:12,
                 from /opt/workspace/build/builtin_interfaces/rosidl_typesupport_opendds_cpp/builtin_interfaces/msg/dds_opendds/duration__type_support.cpp:8:
/opt/workspace/build/builtin_interfaces/rosidl_generator_cpp/builtin_interfaces/msg/detail/duration__struct.hpp:14:10: fatal error: rosidl_runtime_cpp/bounded_vector.hpp: No such file or directory
   14 | #include "rosidl_runtime_cpp/bounded_vector.hpp"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
gmake[2]: *** [CMakeFiles/builtin_interfaces__rosidl_typesupport_opendds_cpp.dir/build.make:136: CMakeFiles/builtin_interfaces__rosidl_typesupport_opendds_cpp.dir/rosidl_typesupport_opendds_cpp/builtin_interfaces/msg/dds_opendds/duration__type_support.cpp.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:307: CMakeFiles/builtin_interfaces__rosidl_typesupport_opendds_cpp.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....
gmake: *** [Makefile:149: all] Error 2
---
Failed   <<< builtin_interfaces [5.47s, exited with code 2]
Aborted  <<< unique_identifier_msgs [7.32s]

Summary: 0 packages finished [8.19s]
  1 package failed: builtin_interfaces
  1 package aborted: unique_identifier_msgs
  2 packages had stderr output: builtin_interfaces unique_identifier_msgs
  7 packages not processed

I might try getting the proper docker image/install setup on a ROS2 humble base image first

I setup this repo, https://github.com/tmayoff/opendds_ros_infra, can run the enter_workspace.sh script and follow the rest of the rmw_build instructions

This error
Error: /__w/opendds_ros_infra/opendds_ros_infra/build/action_msgs/rosidl_typesupport_opendds_cpp/action_msgs/msg/dds_opendds/GoalInfo_C.h:8:10: fatal error: builtin_interfaces/msg/dds_opendds/Time_C.h: No such file or directory
    8 | #include "builtin_interfaces/msg/dds_opendds/Time_C.h"

looks like it could be related to code generation when one idl #includes another.  I'll work on getting the container running locally so I can try to learn more.

The previous error (#include "rosidl_runtime_cpp/bounded_vector.hpp") was solved by adding rosidl_runtime (or something) to the target_link_libraries in one of the files (I forget which now, one of the only changes in the 2 forks I made).

I have the file install/builtin_interfaces/include/builtin_interfaces/msg/dds_opendds/Time_C.h so either that's getting built too late or the relevant -I is missing.

@tmayoff Whats setting are required to make the ROS build infrastructure log the compilation commands?

I'm not sure the command itself but I think it's the log_command event handler documented here. I can add that to the CI runs so it's available to see

This is the latest run with the debug logging and command outputs: https://github.com/tmayoff/opendds_ros_infra/actions/runs/7564009620/job/20597425937

It shows what colcon is running:
DEBUG:colcon.colcon_core.event_handler.log_command:Invoked command in '/__w/opendds_ros_infra/opendds_ros_infra/build/action_msgs' returned '2': [...snip env...] /usr/bin/cmake --build /__w/opendds_ros_infra/opendds_ros_infra/build/action_msgs -- -j4 -l4

I think this is essentially running make on the results of CMake's "Unix Makefiles" generator.  So I think we're looking for a way to see the commands that GNU Make is running.

I'm not sure it can be outputted to the screen, this is from the log file.
events.log
[69.646892] (std_msgs) StdoutLine: {'line': b'[ 50%] \x1b[32mBuilding CXX object CMakeFiles/std_msgs__rosidl_typesupport_opendds_cpp.dir/rosidl_typesupport_opendds_cpp/std_msgs/msg/dds_opendds/header__type_support.cpp.o\x1b[0m\n'}
[69.648123] (std_msgs) StdoutLine: {'line': b'/usr/bin/c++ -DOPENDDS_RAPIDJSON -DOPENDDS_SECURITY -DRCUTILS_ENABLE_FAULT_INJECTION -DROS_PACKAGE_NAME=\\"std_msgs\\" -D__ACE_INLINE__ -Dstd_msgs__rosidl_typesupport_opendds_cpp_EXPORTS -I/opt/workspace/build/std_msgs/rosidl_generator_cpp -I/opt/workspace/build/std_msgs/rosidl_typesupport_opendds_cpp -I/opt/workspace/build/rosidl_typesupport_opendds_cpp -I/opt/workspace/install/rosidl_typesupport_opendds_cpp/include -I/opt/workspace/install/builtin_interfaces/include/builtin_interfaces/msg/dds_opendds -I/opt/workspace/install/builtin_interfaces/include/builtin_interfaces/srv/dds_opendds -I/opt/workspace/install/builtin_interfaces/include/builtin_interfaces/action/dds_opendds -isystem /opt/ros/humble/include/rmw -isystem /opt/ros/humble/include/rosidl_typesupport_interface -isystem /opt/ros/humble/include/rcutils -isystem /opt/ros/humble/include/rosidl_runtime_c -isystem /opt/workspace/install/builtin_interfaces/include/builtin_interfaces -isystem /opt/OpenDDS/tools/rapidjson/include -isystem /usr/local/include/orbsvcs -isystem /opt/ros/humble/include/rosidl_runtime_cpp -isystem /opt/ros/humble/include/fastcdr -isystem /opt/ros/humble/include/rosidl_typesupport_fastrtps_cpp -isystem /opt/ros/humble/include/rosidl_typesupport_fastrtps_c -isystem /opt/ros/humble/include/rosidl_typesupport_introspection_c -isystem /opt/ros/humble/include/rosidl_typesupport_introspection_cpp -g -fPIC -Wall -Wextra -Wpedantic -Wall -Wextra -Wpedantic -std=gnu++14 -MD -MT CMakeFiles/std_msgs__rosidl_typesupport_opendds_cpp.dir/rosidl_typesupport_opendds_cpp/std_msgs/msg/dds_opendds/header__type_support.cpp.o -MF CMakeFiles/std_msgs__rosidl_typesupport_opendds_cpp.dir/rosidl_typesupport_opendds_cpp/std_msgs/msg/dds_opendds/header__type_support.cpp.o.d -o CMakeFiles/std_msgs__rosidl_typesupport_opendds_cpp.dir/rosidl_typesupport_opendds_cpp/std_msgs/msg/dds_opendds/header__type_support.cpp.o -c /opt/workspace/build/std_msgs/rosidl_typesupport_opendds_cpp/std_msgs/msg/dds_opendds/header__type_support.cpp\n'}
[69.742146] (-) TimerEvent: {}
[69.759407] (std_msgs) StderrLine: {'line': b'In file included from \x1b[01m\x1b[K/opt/workspace/build/std_msgs/rosidl_typesupport_opendds_cpp/std_msgs/msg/header__rosidl_typesupport_opendds_cpp.hpp:23\x1b[m\x1b[K,\n'}
[69.759682] (std_msgs) StderrLine: {'line': b'                 from \x1b[01m\x1b[K/opt/workspace/build/std_msgs/rosidl_typesupport_opendds_cpp/std_msgs/msg/dds_opendds/header__type_support.cpp:8\x1b[m\x1b[K:\n'}
[69.759831] (std_msgs) StderrLine: {'line': b'\x1b[01m\x1b[K/opt/workspace/build/std_msgs/rosidl_typesupport_opendds_cpp/std_msgs/msg/dds_opendds/Header_C.h:8:10:\x1b[m\x1b[K \x1b[01;31m\x1b[Kfatal error: \x1b[m\x1b[Kbuiltin_interfaces/msg/dds_opendds/Time_C.h: No such file or directory\n'}
[69.759956] (std_msgs) StderrLine: {'line': b'    8 | #include \x1b[01;31m\x1b[K"builtin_interfaces/msg/dds_opendds/Time_C.h"\x1b[m\x1b[K\n'}
[69.760076] (std_msgs) StderrLine: {'line': b'      |          \x1b[01;31m\x1b[K^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\x1b[m\x1b[K\n'}
[69.760204] (std_msgs) StderrLine: {'line': b'compilation terminated.\n'}

Thanks.
Header_C.h includes builtin_interfaces/msg/dds_opendds/Time_C.h.  This means the command should have -I-I/opt/workspace/install/builtin_interfaces but that is not present.  So one option is to add this include directory.
However, -I/opt/workspace/install/builtin_interfaces/include/builtin_interfaces/msg/dds_opendds is present so another option is to change the include to just Time_C.h instead of builtin_interfaces/msg/dds_opendds/Time_C.h.
Since Time_C.h is generated, I think we can either 1) look at the way it is generated or 2) add the path to influence the compilation.
I'll keep digging.

Some more oddities, generated header for the GoalInfo has these sets of includes
include <cstdint>
#include <string>
#include <utility>
#include "builtin_interfaces/msg/dds_opendds/Time_C.h"
#include "tao/Basic_Types.h"
#include "unique_identifier_msgs/msg/dds_opendds/UUID_C.h"
#include "builtin_interfaces/msg/dds_opendds/Time_C.h"
#include "unique_identifier_msgs/msg/dds_opendds/UUID_C.h"

There's duplicated headers,
When I try to add the --filename-only-includes, to opendds_idl, it generates this
#include <cstdint>
#include <string>
#include <utility>
#include "builtin_interfaces/msg/dds_opendds/Time_C.h"
#include "tao/Basic_Types.h"
#include "unique_identifier_msgs/msg/dds_opendds/UUID_C.h"
#include "Time_C.h"
#include "UUID_C.h"

So, did this work, not work, or just weird?

More just weird. This was a different error than the Time_C.h ones from before I thought I had gotten further with this, but I tried again and for now fails for the previous reasons Time_C.h related.  I tried doing as you suggested changing the include to just Time_C.h but can't as it's including it twice one of which with the full path

These duplicated includes are all in the same file written by opendds_idl?  Which one?  Maybe there's a bug in --filename-only-includes

I think I found the header include issue, locally I ran into an include problem with the idl compilation step, but the CI passed, needs further investigation. These are the changes so far to rosidl_typesupport_opendds, https://github.com/OpenDDS/rosidl_typesupport_opendds/pull/50/files

Okay so that did fix the build. It now crashes when trying to run anything, I added the run_ci_test.sh to CI, for now I can work through these problems, thanks for the help so far!

Okay so I didn't get that much further before getting hit with this series of errors
>>> [rcutils|error_handling.c:108] rcutils_set_error_state()
This error state is being overwritten:

  'Handle's typesupport identifier (rosidl_typesupport_c) is not supported by this library, at ./src/type_support_dispatch.hpp:113'

with this new error message:

  'type support implementation 'rosidl_typesupport_c' does not match 'rosidl_typesupport_opendds_cpp', at /workspaces/opendds_ros_infra/src/rmw_opendds/rmw_opendds_cpp/include/rmw_opendds_cpp/RmwAllocateFree.hpp:84'

rcutils_reset_error() should be called after error handling to avoid this.
<<<

>>> [rcutils|error_handling.c:108] rcutils_set_error_state()
This error state is being overwritten:

  'type support implementation 'rosidl_typesupport_c' does not match 'rosidl_typesupport_opendds_cpp', at /workspaces/opendds_ros_infra/src/rmw_opendds/rmw_opendds_cpp/include/rmw_opendds_cpp/RmwAllocateFree.hpp:84'

with this new error message:

  'DDSPublisher failed, at /workspaces/opendds_ros_infra/src/rmw_opendds/rmw_opendds_cpp/src/rmw_publisher.cpp:128'

rcutils_reset_error() should be called after error handling to avoid this.
<<<


DCOM issues

Running DISM /Online /Cleanup-Image /RestoreHealth
Fails. Writes errors to "C:\Windows\Logs\DISM\dism.log"
Filtered out the GUID's, and none of these show up in CLSID or in DCOM.
{53BFAE52-B167-4E2F-A258-0A37B57FF845}
{85381772-2F23-4807-86AF-8D83A172371C}
{30BC39EF-FA14-42C3-BE77-22C7F1DF4E63}
{E9AC7116-8B6F-4A1A-BC43-94CB7CB944E1}
{5336F940-4DF3-4DE6-BE5A-8F581CC18094}
{648A4F44-3730-4184-90FD-07E283163B0E}
{19EA5405-2914-4836-81BD-ACF2D8EC3588}
{125AFB23-8432-46DF-8071-1F921C7462A9}
{FF324B85-BE21-4749-B2FD-B97B12EC2D74}
{4D36E96C-E325-11CE-BFC1-08002BE10318}
{4D36E968-E325-11CE-BFC1-08002BE10318}
{E0CBF06C-CD8B-4647-BB8A-263B43F0F974}
{4D36E972-E325-11CE-BFC1-08002BE10318}
{92616AE4-2D9D-4660-B5A0-7ED4C56E3D76}
{0D608600-D202-403D-A610-DFAAED0466B5}
{BA930F49-E9F0-4FD2-9A24-34C9351BF7DA}
{6939B893-E7E8-477D-896C-B219834A0E85}
{9F63D0B1-9EF8-4AAF-9F31-24CE8538429D}
{AFA9B532-2BA6-481F-A899-08BA56B6BE35}
{EB5853DF-21BD-4CD2-9B59-59B117BA3937}
Here's the tool where I started to conceptualize that perhaps the product you've been working on is 'allowing remote entry into those systems'
Might not be you, could be Microsoft, or whoever... All I know is, when the SID's or GUID's don't show up, I start to get very concerned as to the reason why.
https://youtu.be/rKKmLwxeXbM
At this time, I'm waiting for ASUS to report back over some driver installation issues I've had. Has nothing to do with DISM++ but I figured I would report my findings with you and hope that there's a good explanation for this GUID's not being able to be 'found' anywhere, and why basic DISM won't work.

This problem is not related to Dism++.
I never find these GUIDs in Dism++'s source code.
Closed.


Aligning Depth Map with LiDAR

Hi, thanks for providing the dataset. I want to use the RGBD and LiDAR UGV data. I've extracted the files from the rosbags. What is the process to align point clouds generated from the depth images with the LiDAR point clouds? I tried to use the LiDAR extrinsic matrix but the alignment looks incorrect. Thanks!

Hi there,
Thanks for the interests in our dataset. The dataset is currently undergoing final submission, which includes us double-checking the calibration results. I'll try the alignment task you mentioned and get back to you.

Hi, just want to check if there's been any update regarding this? Thanks

Hi, thanks for providing the dataset. I want to use the RGBD and LiDAR UGV data. I've extracted the files from the rosbags. What is the process to align point clouds generated from the depth images with the LiDAR point clouds? I tried to use the LiDAR extrinsic matrix but the alignment looks incorrect. Thanks!

Hi, I meet the problem that the point clouds generated from the depth images has too much noises. For example,the surface isn't plain. My research also requires the points cloud from different sensors. I would greatly appreciate any advice or insight you may have on this issue.


[RFC] Winograd NNPACK for the ARM_CPU backend

--with @ajtulloch
We're working on shipping TVM on android for some internal products that uses 3x3 conv heavily. We found that the winograd_nnpack + TVM approach the best for shipping for a broad variety of android devices. Here are the reasons:

We're restricted to AOT compilation and can only ship one model (packed with TVM generated code) to all android devices.
We did autotuning with the direct and winograd implementation on a raspberry pi (Cortex-A53) and found that NNPACK actually outperforms the best autotuned schedules for most of the layers
In the case where AutoTVM does find better schedules on Cortex-A53, the performance does not necessarily transfer to other microarchitectures. Autotuning works best for a fixed CPU microarchitecture. When we ship a model, we care about its performance on wide variety of devices (Cortex-A7, A9, A35, A53, A57, A72, A73, A75, Qualcomm Kryo, Samsung Mongoose M1, M2, and Meerkat M3, to name a few). It is very hard to get a schedule that performs well across the board.

In the end, we decided to use NNPACK Winograd for all 3x3 conv and use TVM for the rest of layers (so we can fuse the layers and parallelize them). This gives us the best overall performances. At the same time, we're building up the infra to target ship model based on the CPU microarchitecture so we can leverage AutoTVM to get the best performance.
Here is our work in progress: https://github.com/hlu1/tvm/tree/winograd-nnpack-ARM. We would like to contribute back if there's interest from the community.

Thanks for proposing it. This will probably serve as a great example of leveraging external libraries in TVM. On the other hand, your observation indicates that within TVM we still have much room to improve the performance. BTW, do you have any performance comparison numbers between AutoTVM and NNPACK on Winograd? Also, does NNPACK use one implementation for a wide variety of devices?

Thanks for the proposal. can you clarify if the winograd AutoTVM is using the template from the mainline? It would be really great if we can also work together to improve the AutoTVM itself for ARM

I did the autotuning (AutoTVM with winograd from TVM master) a while ago. I'm going to redo it to get more up-to-date numbers. NNPACK has two implementations for convolution with winograd transform, wt8x8 and wt8x8_fp16. The fp16 implementation is usually the 20% - 30% faster than the fp32 version. On ARM CPUs with fp16 compute support (only Cortex-A76 I believe), NNPACK would run the microkernel with fp16 compute. On other ARM CPUs with fp16 support but not fp16 compute, NNPACK uses fp16 for intermediate storage only (eg. transformed input and kernel). It'll be hard to beat NNPACK without fp16.

I redid autotuning with the ARM winograd implementation in the master on a fixed frequency raspberry pi cluster (600 MHz, single threaded). The network is a unet with 3x3 convs (https://github.com/hlu1/tvm/blob/nnpack-precompute-ARM/unet/unet.py). Here is the result:

The difference between the nnpack fp16 and fp32 implementation is not that big. The gap between winograd_nnpack and the TVM winograd is almost 2x.

Would be interesting to see if we can improve the tvm's f32 winograd by reusing some of the nnpack's tricks(looking into the ukernel) . In the meanwhile, we can add the nnpack support to the tvm as external lib

PR : #2721

@hlu1 If I understand correctly, In your use case you care about the performance for a variety of microarchitectures, given a single schedule template. If this is the case, we might want a different optimization target for autotvm.

We added template, 'winograd_nnpack', for autotvm, similar to 'direct' and 'winograd'.

One thing that I'm not quite sure of is how to handle fp16. For the 'winograd_fp16' implementation, the transformed weights are stored in fp16 format. For us it's fine because we transform the weights from fp32 to fp16 at runtime with C++. However, the results are incorrect if I test with python.

Hi @hlu1 I have one question. If we use NNPACK FP16, whether it has correctness problem? Because our convolution's input / weight is FP32.

It depends on the application. There is accuracy drop with the winograd_fp16 implementation. We find it acceptable with most of our mobile CV models. It's not uncommon to have fp16 inference with fp32 weights, e.g., Apple's Metal Performance Shader library, which uses fp16 compute on GPU.

@hlu1 In my option, I would suggest we have one option to control whether we turn on tvm.contrib.nnpack.ConvolutionAlgorithm.WT_8x8_FP16 during FP32 computation. Because we don't know what other people's requirement / application. Maybe they can not afford the accuracy drop.
Additionally, I am interested in FP16's performance. Because most of ARM CPU doesn't support FP16 directly (if I remember correctly) like GPU, so ARM CPU will use intrinsic (such as extend to FP32 to compute and truncate it back to FP16) and should have worse performance, however I find it has better performance . The reason is the bandwidth could be half of FP32? Could you help to explain it?

How about having one template winograd_nnpack_fp16 for the fp16 implementation and winograd_nnpack_fp32 for the fp32 one? This way the users will have the option of turning fp16 on/off.
The performance benefit of fp16 comes from:

reduced memory bandwidth
better cache efficiency due to less memory footprint. Fp16 is used for input/output/kernel transforms
For most ARM CPUs that don't support fp16 arithmetic intrinsics, NNPACK uses fp16 only for intermediate storage. It automatically runs the ukernel with fp16 compute on CPUs that do support fp16 intrinsics.

@hlu1 yes. I think winograd_nnpack_fp16 and winograd_nnpack_fp32 is ok. Could you update it in your PR #2721 too?
For NNPACK FP16. if CPU doesn't support FP16 arithmetic, NNPACK will use FP32 to compute (extend FP16 to FP32 and truncate it back to FP16) or else? If it is, I think I could understand FP16 has better performance and why it doesn't beyond FP32 much.

I'll update my PR to reflect this change.
Yes, NNPACK does fp32 compute when fp16 compute is not available (extend FP16 to FP32 and truncate it back to FP16 when computation is complete).


Call Out "Try It Yourself" Tables

Call out the "try it yourself" sections with a table and a title.  Users don't read text, so hiding it there means it is missing when it is actually needed.

Addressed in https://github.com/fprime-community/fprime-workshop-led-blinker/pull/17


Fix(challenges): reset challenge should update main iframe

closes #9956

:sparkles: :sparkles: SWEET. :sparkles: :sparkles:
Thanks.


FluidIdRegistryMessageHandler exception when joining server

When joining my multiplayer server, I get a message saying "A fatal error has occurred, this connection has been terminated."
Here is a snippet from my fml-client-latest.log file: http://pastebin.com/0xh3u6gg
It seems to throw "java.lang.IllegalArgumentException: value already present: veganoption.someliquid" every time I attempt to join the server. Removing the mod gets rid of the problem.
This is present using Forge 1448 on 1.7.10 and Java 1.8.0_40 even when it is the only mod installed.

When running the server on Windows, or in singleplayer, this issue is nonexistent.
It has occurred on my Linux VPS (CentOS) and 2 different shared Minecraft hosts.

Looks like this was caused by https://github.com/MinecraftForge/MinecraftForge/issues/1973
For anyone that runs into this in the future: upgrade to Forge version 1490 or above.


Adding chrome API

Hello,
I would like to add "chrome API" - that is, all the various functions that are available for chrome packaged apps/extensions. Stuff from here - https://developer.chrome.com/extensions/api_index
I don't know if it's better to add that here somewhere (where? it's not a npm package), or to "main" flow into the built-in definitions.
I have a definition file with stuff that I am using here
https://github.com/trezor/trezor-chrome-extension/blob/master/flowtype_definitions/chrome.js
it doesn't cover full API by far, but it's something I want to share anyway

....on the other hand, node definitions are already in flow/lib even when node is not part of "core" javascript, so it's probably better to put it there. Not sure (that's why I am asking :))

If I have some time this evening I will integrate it and PR.
You want to integrate a FlowType for the Chrome API, right?

yep. I now made a PR here
https://github.com/facebook/flow/pull/2094

Mmm okay!
Do you wanna that I made a PR here or you're already doing it?

You can do it, I am not sure where to put it here

Is there any progress on this? I'm currently working on a chrome extension and have written some of the API definitions. I would like contribute to this.

I have no idea. If you want to, you can use my definitions.

Ok, it looks like your definitions contain mostly for Chrome apps. One question though, how should I write tests for this?

yep, I needed that for Chrome apps. It seems like Chrome will cancel chrome apps sooner than this will be merged anywhere :)

I've been using flow-interfaces-chrome for this use case, and the installation is done by adding a line to [libs] in .flowconfig. Until there's a longer term solution for non-NPM libraries, this approach seems reasonable.

I wrote what I needed for my app, not more. Again feel free to use it in
your projects; but I won't be updating it now (especially when Google wants
to kill chrome apps :( )
On Jan 27, 2017 5:50 PM, "Ross Allen"<EMAIL_ADDRESS>wrote:

I've been using flow-interfaces-chrome
https://github.com/Shraymonks/flow-interfaces-chrome/ for this use
case, and the installation is done by adding a line to [libs] in
.flowconfig. Until there's a longer term solution for non-NPM libraries,
this approach seems reasonable.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/flowtype/flow-typed/issues/201#issuecomment-275713504,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGZ8fUxXPIDIMTrEVeVspqDL-Tw-MEDks5rWiBOgaJpZM4JOzVw
.


Update phpunit to version 8

I've updated phpunit to version 8 and fixed all the warning it produces. Once we merge this we can fix failing tests in master and #21 .

Tests failed on 73-lowest?

Yes, it's because guzzle in version below 6.3 has a bug. I'm updating minimum guzzle version which fixes the issue.

I'm sorrty for confusion, I got things mixed up a bit, closing this PR.


Implement Extend<&'a T> for T: Copy

This PR supports the following Vec idiom:
let mut v: Vec<u8> = vec![];
v.extend("data".as_bytes());
assert_eq!(v, [0x64, 0x61, 0x74, 0x61]);

Hey, thanks for the PR. This seems like a pretty straightforward change, and thanks for adding a test. Looks good to me.

Merged. When I'm back at my machine later I'll publish a new version

Merged. When I'm back at my machine later I'll publish a new version

Sweet, may I squeeze #14 in as well before that?


Add RasterCache option to RepaintBoundary widget

Use case
Although RepaintBoundary is a key widget in improving Flutter’s performance on the UI thread, it’s not well-utilized on the GPU thread. For example, only the lowest-level RepaintBoundary, which corresponds to leaf-level engine PictureLayer, currently triggers the raster cache on the GPU thread. In certain scenarios, raster caching can make a big performance difference -- e.g., 20%-30% in our home_scroll_perf average_frame_rasterizer_time_millis benchmark.
With the recent introduction of retained EngineLayers, we now have the opportunity to use RepaintBoundary to trigger more raster caching on the higher level RepaintBoundary. Similar to the recent opacity and PhysicalShapeLayer optimizations, we may be able to improve our performance by multiple times.
Proposal
Add a RasterCache option to the RepaintBoundary widget, so the developer can decide when to engage the optimizations. This option is not only to override our current raster cache heuristics, which are not very smart or reliable, but also allows developers to elect to make the tradeoff between quality and speed themselves. For example, when there’s a rotation transformation above the RepaintBoundary, raster caching the whole RepaintBoundary means that we’re no longer drawing pixel-perfect graphics, but the performance gain may significantly outweigh the quality drop (see https://github.com/flutter/flutter/issues/24627).
Things to implement:

[ ] Write performance tests for the scenarios where we’d like to see improvements. This will guide our implementation and check our correctness and performance.
[ ] Add a RasterCache option to the RepaintBoundary widget and wire it up between the framework and the engine.
[ ] Enable the engine-side raster caching based on the option.
[ ] Document the feature, and update Github issues to reflect the performance improvement, to teach our developers how to do it themselves.

Proposed by @liyuqian

Thanks for polishing the proposal! Closing https://github.com/flutter/flutter/issues/27952 as a duplicate and add a reference to https://github.com/flutter/flutter/issues/27677

Paste Yegor's comment here for convenience: "It sounds a lot like will-change. Wondering if we could just map the proposed RasterCache to it."

Any progress on this?

@flar : I have the same question on the progress :smile:
We just received some internal bug b/137395062 last week, and its current solution cl/260167188 is to manually take a snapshot of the blurred widget. Adding RasterCache option to RepaintBoundary widget would:

Make developer do much less work for these kind of issues. Potentially just one line of change on RepaintBoundary instead of 50 lines of code to snapshot manually.
Make CPU/GPU do much less work as our current snapshot requires GPU->CPU->GPU copies. And according to comments in cl/260167188, Flutter also strangely requires an unnecessary raw->PNG->raw encoding/decoding process. (I'll write a test and probably create a issue on that encoding issue).

I'm looking into revisiting my RepaintBoundary caching work. Meanwhile I think that @chinmaygarde had some suggestions or feedback on faster ways to snapshot widgets that might help with the solution for b/137395062

@liyuqian @flar What is the status of this issue? What is the remaining work to do?

I've been working on various internal caching mechanisms that would work automatically behind the scenes and fixing up errors in our existing caching mechanisms. Since RepaintBoundaries aren't visible to the engine, a significant rework of our layer mechanism would be required to make them visible so that they can control caching.

@flar I just want to know the status of this issue, is it ready for us?

A related suggestion! Rather than a boolean RasterCache option, how about a cacheStrategy option, with CacheStrategy.default being the current behavior of "trying to figure out the best option itself", cacheStrategy.raster for the people wanting that previously in this bug, and for me, CacheStrategy.noCache, which paints the child the normal way on the existing buffer as if RepaintBoundary wasn't present.
The distinction between CacheStrategy.noCache and not having a RepaintBoundary is for the case of non-overlapping widgets, let's say animatingWidget and staticWidget[1-10]. With no RepaintBoundary, all ten staticWidgets repaint every frame because animatingWidget is updating. With a RepaintBoundary on animatingWidget there's a buffer allocated for animatingWidget; the buffer isn't providing anything useful because animatingWidget is repainting every frame anyway, but the RepaintBoundary is preventing the other widgets from repainting when they don't need it. So the RepaintBoundary serves a purpose, but the buffer does not.
The buffer, in this case, incurs a nontrivial memory cost and maybe also a performance cost.

@liyuqian @flar Is there any update on this issue?

As a workaround, I created a widget that attaches cache hints (isComplex, willChange) to the current/parent RepaintBoundary like this:
import 'package:flutter/material.dart';
import 'package:flutter/rendering.dart';

/// set `isComplex`, `willChange` hints to current DisplayList
class RasterCacheHint extends SingleChildRenderObjectWidget {
  const RasterCacheHint({
    Key? key,
    this.isComplex,
    this.willChange,
  }) : super(key: key);

  final bool? isComplex;
  final bool? willChange;

  @override
  void updateRenderObject(
    BuildContext context,
    _RasterCacheHintRenderObject renderObject,
  ) {
    renderObject.isComplex = isComplex ?? false;
    renderObject.willChange = willChange ?? false;
  }

  @override
  RenderObject createRenderObject(BuildContext context) =>
      _RasterCacheHintRenderObject(isComplex, willChange);
}

class _RasterCacheHintRenderObject extends RenderProxyBox {
  _RasterCacheHintRenderObject(bool? _isComplex, bool? _willChange)
      : isComplex = _isComplex ?? false,
        willChange = _willChange ?? false;

  bool isComplex;
  bool willChange;

  @override
  void paint(PaintingContext context, Offset offset) {
    // we need this line to generate a PictureLayer to set the hints
    context.canvas;

    if (isComplex) {
      context.setIsComplexHint();
    }

    if (willChange) {
      context.setWillChangeHint();
    }

    super.paint(context, offset);
  }
}

Hi, I wonder whether the toPictureSync will solve this problem?

The raster cache is a legacy feature at this point (no plans on adding it to impeller) so I'm going to close this.


SwiftCompile task should not analyse Swift source files to extract header file dependencies

The SwiftCompile task reuses a bunch of stuff from AbstractNativeCompileTask to do its thing. This happens to include preprocessor header file dependency analysis. However, the Swift language has no preprocessor and the (Gradle) SwiftCompiler does not use any of the information from the dependency analysis. This is completely pointless work.

@adammurdoch is this probably done now because SwiftCompile and SwiftCompiler no longer share anything with the other native compilation tasks?

Yeah, we should close this. There's still a lot of complexity in there that I think we should remove at some point but we can probably live with this for now.


Update README.mkdn

Thank you for the patch! The README.mkdn file is autogenerated from the documentation in lib/WWW/Mechanize/Chrome.pm , so if you could change that file as well, it would be very much appreciated!


Fix Query with InfluxQL

Fix: {'results': [{'error': 'database name required', 'statement_id': 0}]} when using Influxdb with InfluxQL.
Add Field "db" to query:
{'data': {'refId': 'test', 'db': 'icinga2', 'q': 'SELECT ....'}}
Description
When using grafana-client with influxdb/influxql query is missing db field.
Checklist

[ ] The patch has appropriate test coverage
[ ] The patch follows the style guidelines of this project
[ ] The patch has appropriate comments, particularly in hard-to-understand areas
[ ] The documentation was updated corresponding to the patch
[ ] I have performed a self-review of this patch

Dear @storg2001,
grafana-client 3.11.0 has just been released, including your improvement. Thank you again.
With kind regards,
Andreas.

Thank you


Add authentication endpoints

This PR adds register, login endpoints to the backend. The login endpoint uses JWT for stateless authentication. The endpoints don't talk to the database yet.

I've updated the API routes to reflect usage of JWT for authentication. Btw, do we need both username and email address when registering / logging in?

Let's do just email please

@hmarcks I'll remove the username in a follow-up PR. Merging this now.


Can only take 1 pointlight

only takes the value of the first pointlight and ignores the next

Fixed in #602


Reorganize BertCodeDataset

This just moves functions around so they are closer to their actual usages. Many
are only used in subClasses so the functions are moved into the respective subclasses.

Codecov Report

Merging #1244 (f56baff) into master (0ffff4f) will decrease coverage by 0.04%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #1244      +/-   ##
============================================
- Coverage     70.24%   70.20%   -0.05%
+ Complexity     5068     5067       -1
============================================
  Files           486      486
  Lines         22265    22265
  Branches       2375     2375
============================================
- Hits          15641    15632       -9
- Misses         5413     5423      +10
+ Partials       1211     1210       -1

Impacted Files
Coverage Δ

...gine/src/main/java/ai/djl/dlr/engine/DlrModel.java
51.61% <0.00%> (-3.23%)
:arrow_down:

...rt/src/main/java/ai/djl/tensorrt/jni/LibUtils.java
71.05% <0.00%> (-2.64%)
:arrow_down:

api/src/main/java/ai/djl/util/Platform.java
69.81% <0.00%> (-1.89%)
:arrow_down:

...ain/java/ai/djl/tensorflow/engine/TfNDManager.java
85.71% <0.00%> (-1.69%)
:arrow_down:

api/src/main/java/ai/djl/repository/Artifact.java
87.80% <0.00%> (-1.63%)
:arrow_down:

...ngine/src/main/java/ai/djl/mxnet/jna/LibUtils.java
40.00% <0.00%> (-1.18%)
:arrow_down:

.../djl/training/listener/MemoryTrainingListener.java
76.62% <0.00%> (ø)

...rc/main/java/ai/djl/paddlepaddle/jni/LibUtils.java
48.27% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0ffff4f...f56baff. Read the comment docs.


services/{bridge,compliance}/internal/config: remove mysql support

PR Checklist
PR Structure

[x] This PR has reasonably narrow scope (if not, break it down into smaller PRs).
[x] This PR avoids mixing refactoring changes with feature changes (split into two PRs
otherwise).
[x] This PR's title starts with name of package that is most changed in the PR, ex.
services/friendbot, or all or doc if the changes are broad or impact many
packages.

Thoroughness

[x] ~This PR adds tests for the most critical parts of the new functionality or fixes.~
[x] ~I've updated any docs (developer docs, .md
files, etc... affected by this change). Take a look in the docs folder for a given service,
like this one.~

Release planning

[x] !I've updated the relevant CHANGELOG (here for Horizon) if
needed with deprecations, added features, breaking changes, and DB schema changes.~
[x] ~I've decided if this PR requires a new major/minor version according to
semver, or if it's mainly a patch change. The PR is targeted at the next
release branch if it's not a patch change.~

Summary
Remove MySQL support from config parsing in Bridge and Compliance services.
Goal and scope
Clean up code that is no longer supported. We announced that MySQL was no longer supported for Bridge and Compliance a few months ago in commits 9d0a49bee411 and addf94e97f98 but there was still some code left over in both services to parse configs containing MySQL database connection details. Users of these services who are using MySQL won't see an error when their config is parsed and they may continue to use the database even though we don't support it. It's better that we provide immediate at config parse rather than folks use features we no longer support.
Close #1659
Summary of changes

Add tests for the database type and URL config values for both Bridge and Compliance, because there was no tests for the area of code that needed to change.
Remove MySQL as valid values.

Known limitations & issues
N/A
What shouldn't be reviewed
N/A

Codecov Report

Merging #1725 into master will decrease coverage by 0.08%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #1725      +/-   ##
==========================================
- Coverage   44.86%   44.77%   -0.09%
==========================================
  Files         382      384       +2
  Lines       25984    26103     +119
==========================================
+ Hits        11657    11687      +30
- Misses      13150    13218      +68
- Partials     1177     1198      +21

Impacted Files
Coverage Δ

services/bridge/internal/config/main.go
21.42% <100%> (ø)

services/compliance/internal/config/main.go
28.57% <100%> (ø)

services/horizon/internal/txsub/system.go
71.87% <0%> (+0.78%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4a590dd...a4a03d6. Read the comment docs.

Codecov Report

Merging #1725 into master will decrease coverage by 0.09%.
The diff coverage is 100%.

@@            Coverage Diff            @@
##           master    #1725     +/-   ##
=========================================
- Coverage   44.86%   44.76%   -0.1%
=========================================
  Files         382      384      +2
  Lines       25984    26103    +119
=========================================
+ Hits        11657    11686     +29
- Misses      13150    13220     +70
- Partials     1177     1197     +20

Impacted Files
Coverage Δ

services/bridge/internal/config/main.go
21.42% <100%> (ø)

services/compliance/internal/config/main.go
28.57% <100%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4a590dd...0958e10. Read the comment docs.


feature add Backends table-valued-function

Proposed changes
Issue Number: close #xxx
Problem summary
This pr implement a new Metadata TVF called backends. And the implement process tutorial is in #17974.
The use of backends TVF like:
mysql> select * from backends("cluster_name" = "default")\G
*************************** 1. row ***************************
              BackendId: 10008
                Cluster: default_cluster
                     IP: <IP_ADDRESS>
          HeartbeatPort: 9159
                 BePort: 9169
               HttpPort: 8149
               BrpcPort: 8169
          LastStartTime: 2023-03-20 14:11:09
          LastHeartbeat: 2023-03-20 22:43:44
                  Alive: true
   SystemDecommissioned: false
  ClusterDecommissioned: false
              TabletNum: 31
       DataUsedCapacity: 0
          AvailCapacity:<PHONE_NUMBER>21
          TotalCapacity:<PHONE_NUMBER>064
                UsedPct: 74.409744379782083
         MaxDiskUsedPct: 74.409744379813631
     RemoteUsedCapacity: 0
                    Tag: {"location" : "default"}
                 ErrMsg:
                Version: doris-0.0.0-trunk-e71f7b5066
                 Status: {"lastSuccessReportTabletsTime":"2023-03-20 22:43:02","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false}
HeartbeatFailureCounter: 0
               NodeRole: mix
1 row in set (0.07 sec)

Checklist(Required)

[ ] Does it affect the original behavior
[ ] Has unit tests been added
[ ] Has document been added or modified
[ ] Does it need to update dependencies
[ ] Is this PR support rollback (If NO, please explain WHY)

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run beut

run p0

run buildall

run buildall

run p0


Multi-Host Scaling

closes #13

Code works as seen in the run on wandb. Next, we should discuss whether we have number_of_hosts runs on wandb, but I couldn't figure out how to synchronise wandb configs yet.


iplb module running with l2_learning

Hello,

VM1, VM2, h1, h2 and h3 are all connected to same network <<IP_ADDRESS>/24>. VM1 and VM2 are pingable with s1 running l2_learning module [VM2's physical interface added to s1].
I am generating traffic from VM1 targeting h1 but s2 [connected to iplb load balancing module] is not directing traffic to h1 [s1 is the source of traffic for s2, s2 receives and processes first few incoming packets and then stops].

What changes should I make to the load balancer in order to be able to properly process packets  coming from s1.
Any comments or suggestions would be great. Thanks.

The first thing I'd do is examine the traffic between s2 and one of the hosts.  Based on what you said and the log messages, it seems likely that the start packets of some TCP connections are making it to the hosts, so you should be able to see those (the "SYN" packets of the TCP three-way handshake).  So are the hosts accepting the packet?  One way of answering that would be looking at the packets going from the host back to s2.  If you see the SYN+ACK packets from the TCP handshake, that's probably a good thing.  Another possibility is that you'll see ARP packets ARPing for the source or for a gateway or something.  In which case you'll hopefully see ARP replies too.  In short, since it seems like things may be working part of the way, this is just attempting to figure out the point at which things stop working.
You could also try to see what the host itself thinks is happening.  For example, checking its ARP table to see if it has complete entries, seeing if the TCP server involved is seeing connections, etc.
Another possible approach would be to simplify things a bit and see if that works.  For example, if you hook another host/VM directly up to s1 (without the "NAT Internal Network" being involved), can it communicate with the hosts as expected?

No activity for a while, so I'm going to close this with the hope that it got sorted out.


How to separate encrypted and evaluator

The server received the ciphertext, public and relin keys. How does the server calculate the value without context? My current understanding is that if we want to calculate the ciphertext, we need Evaluator, but Evaluator depends on context. If we transmit the context, The server can generate its own private key based on the context, which may lead to user information leakage.

You are right that you only need the ciphertext, the circuit and the public and relin keys. I don't believe the context contains the secret key. It just contains all the necessary security parameters to be able to create the correct circuit. I believe the secret key is managed by the KeyGenerator class.

The transmission of a Ciphertext to another server for homomorphic evaluations requires the server to use the same encryption parameters that were used to originally create the Ciphertext. Optionally, the server would also need a public key to be able to encrypt Plaintexts, a relin key for relinearization, or a galois key for rotations.
You wouldn’t transmit the original Context (there’s no serialization method), but instead the EncryptionParameters (which has a serialization method). The server would then create its own Context from those parameters.
From there, the server can generate a KeyGenerator, but it would not be able to create the same secret key.