Update http4s-circe, http4s-dsl, ... to 0.23.24

About this PR
📦 Updates

org.http4s:http4s-circe
org.http4s:http4s-dsl
org.http4s:http4s-ember-server

from 0.23.23 to 0.23.24
📜 GitHub Release Notes - Version Diff
Usage
✅ Please merge!
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

⚙ Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.http4s" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "30 days" },
  dependency = { groupId = "org.http4s" }
}]

labels: library-update, early-semver-minor, semver-spec-patch, version-scheme:early-semver, commit-count:1

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (fdb6da5) 50.00% compared to head (af0322f) 50.00%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff            @@
##           develop     #490   +/-   ##
========================================
  Coverage    50.00%   50.00%
========================================
  Files            5        5
  Lines           84       84
========================================
  Hits            42       42
  Misses          42       42

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Document working with counters in Flux

There are special ways to handle counter-formatted data in Flux. We need to create a guide that walks through high-level concepts of extracting useful information of counters and how to handle counter resets.
Related Issue: https://github.com/influxdata/flux/issues/2203

Closing in favor of #2931


AttributeError: 'AnonymousUser' object has no attribute 'timezone'

Hi Miserlou,
I'm using your package to auto detect timezone.
I have a problem with trying to save the detected timezone into my user model.
Basically I have a field 'timezone' inside my customized user model , but I cannot access it to save  unless user have to logged-in first?!
The 'django-easy-timezones' provide instance but it is 'AnonymousUser' only:
detected_timezone.send(sender=get_user_model(), instance=request.user, timezone=tz)
(line 88 middleware.py)
My error was:
'AnonymousUser' object has no attribute 'timezone'
How do I modify 'django-easy-timezones' to work only for logged user only ?
I'm using default signals in my views.py:
"from easy_timezones.signals import detected_timezone
@receiver(detected_timezone, sender=MyUserModel)
def process_timezone(sender, instance, timezone, **kwargs):
if instance.timezone != timezone:
instance.timezone = timezone
instance.save()"

nvm ... check instance is_authenticated()


fix(TDI-46222): fix CVE-2021-29425

Update commons-io to 2.8.0

What is the current behavior? (You can also link to an open issue here)
What is the new behavior?
Please check if the PR fulfills these requirements

[ ] The commit message follows Talend standard
[ ] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features) ?
[ ] The code coverage on new code >75%
[ ] The new code does not introduce new technical issues (sonar / eslint)

What kind of change does this PR introduce?

[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build / CI related changes
[ ] Other... Please describe:

Does this PR introduce a breaking change?

[ ] Yes
[ ] No

If this PR contains a breaking change, please describe the impact and migration path for existing applications: ...
Other information:
https://jira.talendforge.org/browse/TDI-46222

Pull request test result: failure :x:


Auto detect branch and repo for build badge (Travis CI)

Badge status is now based on latest build irrespective of branch. If latest build fails, the badge shows failing on every branch.
One solution could be pointing the badge link to either master or development. But, this always shows the status of pointed branch and requires the badge link to be updated in each branch checked out. Same scenario even if its forked.
Refrence issue #1892

#114 Wrongly raised due to github server issue.

#114 Wrongly raised due to github server issue.


Fix for issue #2276

This should fix the issue reported in https://github.com/fchollet/keras/issues/2276

Now enforcing a MaxNorm constraint on the embeddings seems to work properly - here's a quick test on a text classification task:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import numpy as np

from keras.preprocessing import sequence
from keras.models import Sequential
from keras.layers.core import Dense, Activation, Flatten
from keras.layers.embeddings import Embedding
from keras.constraints import MaxNorm
from keras.datasets import imdb

(X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=500, test_split=0.05)
X_test = sequence.pad_sequences(X_test, maxlen=100)

threshold = 0.01
for norm_constraint in [None, MaxNorm(m=0.01, axis=1)]:
    np.random.seed(1337)

    model = Sequential()

    embedding_layer = Embedding(500, 10, input_length=100, W_constraint=norm_constraint)
    model.add(embedding_layer)
    model.add(Flatten())
    model.add(Dense(1))
    model.add(Activation('sigmoid'))

    model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
    model.fit(X_test, y_test, batch_size=32, nb_epoch=5, validation_data=(X_test, y_test), verbose=2)

    W = embedding_layer.trainable_weights[0].get_value()

    if norm_constraint is not None:
        for i in range(500):
            assert np.linalg.norm(W[i, :]) - threshold < 0

Right, thanks for the fix.


AutoMapper - DLL Not put in Bin TFS and MSBuild

I have a Web Application Project that separates my "Host" into a Sub DLL for re-use purposes.
Web Application Project -> Host (Configure, IdenityServer3, Controllers etc).
The Host Project is using IdentityServer3.EntityFramework, which relies on AutoMapper. I recently upgraded to IdentityServer3.EntityFramework.2.4.2 and of course AutoMapper 4.2.1.
My Web Application Project does not include the nuget packages from the above 2.
When I compile with Visual Studio, the AutoMapper.dll gets put into the bin directory. However, when we check-in to TFS and use msbuild to target the WebApplication, then AutoMapper does not get included in the bin directory.
I have 2 choices.

Add a reference in my WebApplication to AutoMapper.
In my Host Project, add a line of code like this.
        AutoMapper.IMapper _doNotRemoveHack; // MSBuild is not pulling Automapper.dll

I do not want to add a reference, cause that makes no sense when that Project doesn't need it. I also hate the hack as well, but I took option 2 and that resolved my issue.
This is an issue / difference with VS and MSBuild; however I wanted to let you know about this.
I didn't have this issue until I upgraded.

Thanks for the note. I'm really not sure how others handle this, but this will be here in the archives for anyone else. Thx.


Powershell Install-dbaMaintenanceSolution parameters don't work

When using the powershell module Install-dbaMaintenanceSolution cleanuptime and backup drives parameters do not populate in the job scripts
SQL Server 2017 14.0.1000.169
Version of the script
Downloaded from dba-tools
What command are you executing?
Install-DbaMaintenanceSolution -SqlInstance localhost -Database Server_Info -BackupLocation "C:" -CleanupTime 72 -InstallJobs -ReplaceExisting
What output are you getting?
[09:17:10][Install-DbaMaintenanceSolution] Installing on server localhost, database Server_Info.
[09:17:10][Install-DbaMaintenanceSolution] Installing MaintenanceSolution.sql.
[09:17:10][Install-DbaMaintenanceSolution] Installation complete.

Please complete the bug template and provide all details requested.

Bug report link


WoW Pet Import Search

This one is the same as the FFXIV reselling search, the idea is to buy on other selected servers and sell on the home server.
All the ids and realm names should already be in the FE.  We should have the home realm default to whats in the settings, but allow them to change it to whatever they want.
post:
{
    "region": "NA",
    "homeRealmID": 3678,
    "connectedRealmIDs": {
        "5": "Proudmoore",
        "9": "Kil'jaeden",
        "11": "Tichondrius",
        "57": "Illidan",
        "60": "Stormrage",
        "3683": "Dalaran",
        "3684": "Mal'Ganis",
        "3693": "Kel'Thuzad",
        "3725": "Gundrak"
    },
    "ROI": 50,
    "profitAmount": 2000,
    "maxPurchasePrice": 2000,
    "salesPerDay": 20,
    "avgPrice": 1000,
    "includeCategories": [],
    "excludeCategories": [],
    "sortBy": "lowestPrice"
}

output
{
    "data": [
        {
            "ROI": 86,
            "avgPriceTSM": 6192.41,
            "itemID": 2698,
            "itemName": "Prismatic Softshell",
            "link": "https://undermine.exchange/#us-illidan/82800-2698",
            "lowestPrice": 911.87,
            "lowestPriceRealmID": 57,
            "lowestPriceRealmName": "Illidan",
            "minPrice": 9698.66,
            "profitAmount": 8347.45,
            "salesPerDay": 38
        },
        {
            "ROI": 54,
            "avgPriceTSM": 1702.94,
            "itemID": 3331,
            "itemName": "Emmah",
            "link": "https://undermine.exchange/#us-illidan/82800-3331",
            "lowestPrice": 1900.0,
            "lowestPriceRealmID": 57,
            "lowestPriceRealmName": "Illidan",
            "minPrice": 4444.35,
            "profitAmount": 2417.13,
            "salesPerDay": 24
        }
    ]
}

@cohenaj194 What is the api endpoint for this one?


chore: Added Artifact Hub badge to readme

Part of #9769
Signed-off-by: Eddie Knight<EMAIL_ADDRESS>Please do not open a pull request until you have checked ALL of these:

[x] Create the PR as draft .
[x] Run make pre-commit -B to fix codegen and lint problems.
[x] Sign-off your commits (otherwise the DCO check will fail).

[x] "Fixes #" is in both the PR title (for release notes) and this description (to automatically link and close the issue).
[x] Add unit or e2e tests. Say how you tested your changes. If you changed the UI, attach screenshots.
[ ] Github checks are green.
[ ] Once required tests have passed, mark your PR "Ready for review".

If changes were requested, and you've made them, dismiss the review to get it reviewed again.

Got an e2e check failure that I could diagnose... re-pushed to restart the checks

Different failure this time.
Assuming these changes are welcome, could I get a recommendation from a reviewer to help make my checks to go green?


The update process is not compatible with tasks renaming directories

I thought I needed #287 (an additional operation environment variable to decide whether to run things in a task or not), but now that I think about it, it seems to me that the main issue is that during update, the new version of the template is directly rendered in the current project.
Example:

user generates Project from Template v1
copier renders folders and files in Project
copier runs Template v1 tasks
task renames DirA into DirB in Project
developer tags Template v2
user updates Project to use Template v2
copier re-renders Template v1 in temp dir
copier runs Template v1 tasks, so DirA is renamed DirB
copier gets the diff between temp dir and Project
diff stores possible changes in DirB, and none in DirA
copier renders Template v2 in Project
copier creates DirA in any case
copier does not do anything with DirB
copier runs tasks in Project again
two scenarios now

task always runs

task tries to rename DirA into DirB: depending on the instruction, it fails (directory exists and is not empty), it moves DirA inside of DirB, which is wrong, or it recursively overwrites DirB without asking, or more complex stuff like merging directories...
copier applies diff (it could work, it could fail, what happens if it fails?)

task uses OPERATION env var to separate copy from update instructions

task does not try to rename DirA into DirB
copier applies diff
user ends up with both DirA and DirB

What bothers me here, is that the task of dealing with renamed directories and their contents is left to the user. It is unexpected, hard to reason about, and error-prone. The fact that, if we implement something like #287, files or directories that were meant to be renamed could end up not renamed is confusing as well: "the tasks initially renamed this directory, so they will not try to rename it again during an update, but then why the not-renamed directory appears after the update?". I think this confusion is what led me to open #196.
Suggestion:
I think we could improve this process, or better, simplify it. Instead of rendering the new version in-place, asking confirmation for each item before rendering the next, we could, just like for the current/old version of the template, render the new version in a temporary directory, and once it's done (and tasks ran in the new version), recursively "merge" the new version into the current project, asking confirmation when needed.
It would give us the following process:

user generates Project from Template v1
copier renders folders and files in Project
copier runs Template v1 tasks
task renames DirA into DirB in Project
developer tags Template v2
user updates Project to use Template v2
copier re-renders Template v1 in temp dir
copier runs Template v1 tasks, so DirA is renamed DirB
copier gets the diff between temp dir and Project
diff stores possible changes in DirB, and none in DirA (<- same process until here)
copier renders Template v2 in temp dir
copier runs Template v2 tasks, so DirA is renamed DirB
now copier recursively merges temp dir v2 into Project
DirA appears nowhere, so is never recreated
user/developer does not even need "update" tasks!

Interesting... I have some questions:

Did you actually test the wrong workflow or is it just theorical?
What's the purpose of dir rename?
Would you be able to open a PR just adding the test case that fails? I think the case is complex enough and analyzing the test case would provide satisfactory evidence on it and allow us to fix it using TDD.

I did, I noticed and investigated this behavior in a template I write for work.
The directory name should be templated, but I'm using extra context, computed in a context file, so the directory name could be [% from "context" import ctx with context %][[ ctx.directory_name ]], but double-quotes are not valid characters on Windows, so I have no other choice but give it a basic name and rename it in a task. An other solution would be for copier to provide a way to modify the context before rendering.
Of course, I can do that!

Great! I was gonna ask you about using single quotes too.
Then I think the next step should be the tests PR, to be able to reproduce the problem and then think of the fix. Would you mind please?
Thanks!


You removed support for init via raw value

How am I meant to initialize from a number raw value now?

nvm, simply can get the index:
nvLoadingIconTypeIndex = NVActivityIndicatorType.allCases.index(of: NVActivityIndicatorHelper.Constants.type)!
and init from that index:
NVActivityIndicatorHelper.Constants.type = NVActivityIndicatorType.allCases[nvLoadingIconTypeIndex]

Any reasons you have to use a raw number instead of a enum case?

Yes, I so i can change the loading icon remotely using firebase remote config (without updating the app), and that only allows very primitive types like strings and numbers etc.

Interesting! I think we can close this issue as you already have the solution for it.


rsync-local/config.yaml: Make external_devices more permissive

/dev/hd, /dev/sd and /dev/vd are all valid prefixes, with letters a-z, and potentially more than 9 partitions too...
Ping https://github.com/Poeschl/Hassio-Addons/issues/381

Just allowing the definition of them will add none of them to the devices list of the addon.
So Home Assistant will not mount them into the addon-container.
I will not merge this PR, but will do a similar change ;)


fix: settings pane overflowing when number of pages increase

Description
Settings pane was overflowing when the number of pages increases. This PR address the overflow issue.
PR fixes following issue(s)
Fixes https://github.com/appsmithorg/appsmith/issues/28710
Type of change

Bug fix (non-breaking change which fixes an issue)

Testing

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Also list any relevant details for your test configuration.
Delete anything that is not relevant

[ ] Manual
[ ] JUnit
[ ] Jest
[ ] Cypress

Test Plan

Add Testsmith test cases links that relate to this PR

Issues raised during DP testing

Link issues raised during DP testing for better visiblity and tracking (copy link from comments dropped on this PR)

Checklist:
Dev activity

[ ] My code follows the style guidelines of this project
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] PR is being merged under a feature flag

QA activity:

[ ] Speedbreak features have been covered
[ ] Test plan covers all impacted features and areas of interest
[ ] Test plan has been peer reviewed by project stakeholders and other QA members
[ ] Manually tested functionality on DP
[ ] We had an implementation alignment call with stakeholders post QA Round 2
[ ] Cypress test cases have been added and approved by SDET/manual QA
[ ] Added Test Plan Approved label after Cypress tests were reviewed
[ ] Added Test Plan Approved label after JUnit tests were reviewed

/build-deploy-preview skip-test=true

/ok-to-test<EMAIL_ADDRESS>@tag.IDE"

/ok-to-test<EMAIL_ADDRESS>@tag.IDE"


ui: Improvements after system-wide changes

[x]  Decide on the Agent's web UI features / troubleshooting helpers (e.g. pprof generation)
[ ]  Create issue for UI's filtering with lots of data

#629

deprecated


Selected line unreadable in subpage wizard

Describe the bug
Our application contains multiple wizards which have been converted to AL without problems. But when we step through the wizards some of the subpages exhibit peculair behaviour; the selected line in the ListPart is unreadable. It seems as if there is a different line superimposed on the selected line. See first and second screenshots for example.
To Reproduce
See AL Code
AL Code to reproduce the issue

A wizard page with (at least) 2 subpages, see third screenshot.
The content on subpage2 is dependant on the choice made on subpage1, see fourth and fifth screenshot.
Removal of the CurrPage.Update() in the fifth screenshot solves the readability problem, but then a second different choice on subpage1 doesn't update subpage2.
After much trial and error we find that a Reset(), see screenshot 6, solves the problem, but we do not understand why.

Expected behavior
Readable selection with code that had no issues with earlier (C/AL) versions
Screenshots
1.

2.

3.

4.

5.

6.

5. Versions:

AL Language: v5.0.320540

Business Central: NL Business Central 16.4 Mysolution System Application 16.1.20247.9 (Platform 16.0.14648.15420 + Application 14.9.39277)

looks like one of the placeholder lines shown when a list box has been rendered but the data not loaded into it yet

@MOomenMYS can you share here the source code for a simple self-contained project that reproduces the issue?

@qutreson Attached you will find a project. But the strangest thing is that the problem does not occur here? While the only changes I've made are changing the sourcetables to match default tables Customer and Customer Bank Account instead of Resource and our own Project Resource.
ALProject8.zip

Hi! In the last months, we've been working hard on adding new features that increase your productivity and on improving the quality of the tools. Unfortunately, this meant that we were not able to follow up on as many GitHub issues as we would have liked.
If this is still a problem in the latest insiders build, feel free to raise this issue again.


no JS stacktrace in dev?

When an error is logged in Sentry dashboard. We don't get to see the JS stacktrace. See example:

It looks like the source maps aren't getting uploaded.
The Bundle React Native code and images is updated and enabled:

The Upload Debug Symbols to Sentry build phases is enabled:

And I see the iOS debug symbol upload success everytime i build using xcode.
Based on this doc: https://github.com/getsentry/react-native-sentry/blob/master/docs/sourcemaps.rst
The source maps should be automatically generated (and uploaded?)
Am I missing something?

Is this the reason the sourcemaps isnt getting uploaded?

@vafada That message is from react-native.
We disabled the sourcemap upload by default for simulator builds. The generated script is always changing and we figured it does not make much sense always uploading it.
But you can enable it if you want, it's a bit hidden in the documentation but if you pass –allow-fetch.
../node_modules/sentry-cli-binary/bin/sentry-cli react-native-xcode --allow-fetch ../node_modules/react-native/packager/react-native-xcode.sh


[Suffix] middle layer name question service change

Description
Added suffix to various services regarding name question, which allows name question service to take suffix.  ApplicantData#getApplicantNameSuffix will first try to read it from the ApplicantModel (the PAI column), and then fall back to reading it from the WellKnownPath.

Added suffix as one of the WellKnownPaths;
Made changes to ApplicantData, NameQuestion, QuestionAnswer, Scalar service to take suffix;
Added suffix Jr. to QuestionJsonSampler;
Unit tests changes.

Checklist
General
Read the full guidelines for PRs here

[x] Added the correct label: < feature | enhancement | bug | under-development | dependencies | infrastructure | ignore-for-release | database >
[x] Assigned to a specific person, civiform/developers, or a more specific round-robin list
[x] Added an additional reviewer from outside your organization as FYI (if the primary reviewer is in the same organization as you)
[x] Created unit and/or browser tests which fail without the change (if possible)
[x] Performed manual testing (Chrome and Firefox if it includes front-end changes)

Issue(s) this completes
Part of #8267

Approving with the one change to the csv export (that we'll also want in the json export)

change to the csv report is done! will also do that to json export when working on JsonExport service changes.


[doc] Update spark streaming read change log

Purpose

Update spark streaming read change log doc

Tests

API and Format

Documentation

+1


Testing the normal map rendering

Hi! Thanks for the great work. I have ran your code and tried to test the normal rendering using render_mesh functions. Since they are used for training I was wondering how they work and their output, however I am not sure if I render correctly the normal map of a simple cube.
Here is the normal map of the cube I generated using your code:

Here is the normal map of the cube on Blender:

As you can see, it seems that the different faces of the cube are merged together in the normal map I have generated using your code. What do you think of this issue? Is it the behavior intended or I missed something in the code?

Hi, could you provide your code for rendering the cube normal as well as the mesh so that I can make a test?

Yes sure! I am using trimesh for obtaining the faces and vertices as a numpy list. Here is my code for rendering the cube, feel free to let me know if I misunderstood something is the usage of your model:
import numpy as np
import logging
import trimesh
from PIL import Image
from ..models.lrm_mesh import InstantMesh
import importlib
from omegaconf import OmegaConf
import torch
import os

def render_cube():
    # Loading the model
    config_path = "configs/instant-mesh-large.yaml"
    config = OmegaConf.load(config_path)
    model_config = config.model_config
    device = torch.device("cuda")
    model = InstantMesh(**model_config.get("params", dict()))
    model = model.to(device)
    model.init_flexicubes_geometry(device)
    model.eval()

    # Loading the cube
    curr_dir = os.path.dirname(os.path.abspath(__file__))
    obj_path = os.path.join(curr_dir, "cube.obj")
    cube = trimesh.load_mesh(obj_path)

    # Camera pose (hardcoded from blender)
    pose = np.array(
        [
            [1.00000000e00, 0.00000000e00, 0.00000000e00, 0.00000000e00],
            [0.00000000e00, 3.42285418e-08, -9.99999940e-01, -9.36579132e00],
            [0.00000000e00, 9.99999940e-01, 3.42285418e-08, 0.00000000e00],
            [0.00000000e00, 0.00000000e00, 0.00000000e00, 1.00000000e00],
        ]
    )

    # Getting the faces and vertices from trimesh
    faces = cube.faces
    vertices = cube.vertices

    logging.debug(f"Faces shape: {faces.shape}")
    logging.debug(f"Vertices shape: {vertices.shape}")

    # Format the vertices and faces for usage in the model render_mesh function
    vertices = np.array([vertices])
    faces = np.array([faces])
    pose = torch.from_numpy(pose).float()
    pose = torch.linalg.inv(pose).float()
    cameras = np.array(
        [[pose]]
    )  # We have only one camera because one mesh for the cube object

    vertices = torch.from_numpy(vertices).to(device).type(torch.float32)
    faces = torch.from_numpy(faces).to(device)

    logging.debug(f"Vertices shape: {vertices.shape}")
    logging.debug(f"Faces shape: {faces.shape}")
    logging.debug(f"Cameras shape: {cameras.shape}")

    # Rendering the mesh using the model render_mesh function
    mask, hard_mask, text_pos, depth, normal = model.render_mesh(
        vertices, faces, cameras, render_size=512
    )
    logging.debug(f"Normal shape: {normal.shape}")
    logging.debug(f"Depth shape: {depth.shape}")

    # Saving the depth and normal maps
    os.makedirs("output", exist_ok=True)
    output_dir = "output"
    depth = depth[0].cpu()
    normal = normal[0].cpu()
    mask = mask[0].cpu()
    hard_mask = hard_mask[0].cpu()

    depth_path = os.path.join(output_dir, "depth.png")
    normal_path = os.path.join(output_dir, "normal_map.png")
    mask_path = os.path.join(output_dir, "mask.png")
    hard_mask_path = os.path.join(output_dir, "hard_mask.png")

    depth_np = depth.squeeze().numpy()
    normal_np = normal.squeeze().numpy()
    hard_mask_np = hard_mask.squeeze().numpy()
    mask_np = mask.squeeze().numpy()

    depth_np = (depth_np * 255).astype(np.uint8)
    normal_np = (normal_np * 255).astype(np.uint8)
    mask_np = (mask_np * 255).astype(np.uint8)
    hard_mask_np = (hard_mask_np * 255).astype(np.uint8)

    Image.fromarray(depth_np).save(depth_path)
    Image.fromarray(normal_np).save(normal_path)
    Image.fromarray(mask_np).save(mask_path)
    Image.fromarray(hard_mask_np).save(hard_mask_path)

I am also giving you the OBJ file in case it is useful:
# Blender 4.1.1
# www.blender.org
o Cube
v 0.024682 -1.706954 0.292741
v 1.413998 -0.689654 -0.724559
v -1.413998 -0.724559 -0.689654
v -0.024682 0.292741 -1.706954
v 0.024682 -0.292741 1.706954
v 1.413998 0.724559 0.689654
v -1.413998 0.689654 0.724559
v -0.024682 1.706954 -0.292741
s 0
f 1 5 7 3
f 4 3 7 8
f 8 7 5 6
f 6 2 4 8
f 2 1 3 4
f 6 5 1 2

I have done some research and maybe the cause is the interpolation of the normal map when you are splatting face normals to vertices with this code in the neural_renderer.py
def compute_vertex_normal(v_pos, t_pos_idx):
    i0 = t_pos_idx[:, 0]
    i1 = t_pos_idx[:, 1]
    i2 = t_pos_idx[:, 2]

    v0 = v_pos[i0, :]
    v1 = v_pos[i1, :]
    v2 = v_pos[i2, :]

    face_normals = torch.cross(v1 - v0, v2 - v0)

    # Splat face normals to vertices
    v_nrm = torch.zeros_like(v_pos)
    v_nrm.scatter_add_(0, i0[:, None].repeat(1, 3), face_normals)
    v_nrm.scatter_add_(0, i1[:, None].repeat(1, 3), face_normals)
    v_nrm.scatter_add_(0, i2[:, None].repeat(1, 3), face_normals)

    # Normalize, replace zero (degenerated) normals with some default value
    v_nrm = torch.where(
        dot(v_nrm, v_nrm) > 1e-20, v_nrm, torch.as_tensor([0.0, 0.0, 1.0]).to(v_nrm)
    )
    v_nrm = F.normalize(v_nrm, dim=1)
    assert torch.all(torch.isfinite(v_nrm))

    return v_nrm

I am wondering if it is expected to have this kind of smoothing or not. By the way I have tested on a plane and the rendering works fine with a single color

@Kev1MSL  can you provide the code or the way you used to create the normal map of the cube on Blender:


Remove tauri

Will this PR still allow for running the interface via docker and yarn serve?

bump @mamiksik


Add external schema mappings for files written with name-based schemas

Files written by Iceberg writers contain Iceberg field IDs that are used for column projection. Iceberg doesn't currently support tracking data files that were written by other systems and added to Iceberg tables with the API because the field IDs are missing. To support files written by non-Iceberg writers, Iceberg could support a table-level mapping from a source schema to Iceberg IDs.
For example, a table with 2 columns might have an Avro schema mapping like this one, encoded as JSON in table properties:
[ {"field-id": 1, "names": ["id"]},
  {"field-id": 2, "names": ["data"]} ]

When reading an Avro file, the read schema would be produced using the file's schema and the field IDs from the mapping. The names in each field mapping is a list to handle aliasing.

@YuvalItzchakov are you still working on this? If not I can take this up.

@rdblue, I spent some time today looking into it. I have a few questions

Is this feature only meant for Avro datasets/files?
Why do we need to store a table level mapping when the table's [Iceberg] schema may already have field ids mapped to names?

This is for mapping external schemas into Iceberg. Basically adding Iceberg's IDs during conversion. This could be done for any format. There's code for "fallback" in Parquet that does this using the column ordinal, which is how we read old Parquet data.
This particular issue is to add a mapping for Avro, which uses names to resolve columns. This is useful for data coming from an external writer that can write Avro schemas, but doesn't have an Iceberg schema.

Why do we need to store a table level mapping when the table's [Iceberg] schema may already have field ids mapped to names?

Iceberg names are not necessarily the names used by the writer. Avro coming from Kafka, for example, allows renaming a column. You can still read older data because Avro contains an Alias for it. If we only used Iceberg's current name for a column, a change like this would cause Iceberg to start ignoring records.

Thanks @rdblue  for that information. Here's a rough spec of what we can do :-

Update Avro.ReadBuilder to take in a Map of field names to fields ids which is passed on to ProjectionDatumReader.
In ProjectionDatumReader if the passed in fileSchema does not have any field ids, we use a modified SchemaToType class to assign ids to fields with the help of the mapping.
We'll need to update SchemaToType to take in a function which can either allocate increasing ids (current default behavior) or assign ids using the map.
Reconvert the returned iceberg schema (SchemaToType returns Type) to Avro using AvroSchemaUtil, which can then be passed to AvroSchemaUtil#buildAvroProjection.

Updating SchemaToType
In order to write a default mapping function to emulate current behavior in SchemaToType. I need to clear up some doubts around how are we currently allocating ids.
In SchemaToType, when called with table schema or record - their top level fields are assigned ids starting from 0, when called with other type -- map, list etc, ids are assigned starting from 1. We can possibly write in a function to emulate the current behavior, or do u see scope here to simplify and only assign ever increasing ids, without caring about whether the top level fields of a schema get ids starting from 0.
For the new function which uses a map to assign ids to fields, I presume for other types - map, list etc we can use non conflicting increasing ids to assign to each field in these container types.
Possible Alternative
Another possible alternative here could be that we don't modify SchemaToType , but use it nonetheless to generate an Iceberg schema.  We define a new method [assignIds] under TypeUtil similar to TypeUtil#assignFreshIds(Schema, TypeUtil.NextID) which can take that schema and a function [similar to NextID] which can assign ids using the mapping.  We will basically be overriding all the ids which SchemaToType would have allocated. I think there's scope of reusing com.netflix.iceberg.types.AssignFreshIds and making it more generic to serve two usecases - assigning fresh ids and assigning ids from a map.  [For non record container types we'd still have to use non conflicting ids to assign to their fields].

Hey @rdblue, do you have any feedback regarding either of the above approaches?

@rdsr, sorry for the delay.
I was thinking about a solution more like the alternative you proposed, but I was thinking that this would work using just Avro schemas, so no need to convert from Iceberg to Avro.
Iceberg already has field IDs, the question is how to match those up with the Avro schema in a data file. We also don't want to change the schema from the file too much because it is required to correctly read the data. So converting to Iceberg, then back to Avro is much more risky than transforming Avro to Avro+ids.
I like your idea to have a some mapping callback, similar to NextID but that can be implemented by something that contains the mapping. Hopefully, that API could be used for Avro and Parquet ID mapping since we'd like to be able to do the same for parquet-avro files, eventually.

@rdblue . Thanks for the input. I see your point. Converting from Avro to Iceberg and then to Avro again may end up possibly losing some information.  In that case, I think we cannot even use SchemaToType as described in the first approach.
I'll see if I can use my second approach but only using Avro to Avro transformations.


All merging of RenderableNodes, such that we can merge multiple topologies.

With this change, the containers topology shows containers not actively communicating.
Fixes #171

@peterbourgon Early WIP, seems to work.

OK, assuming the stuff we chatted about is for a subsequent PR, and after those things are fixed, LGTM.


LE and BE will modify the order of qubit register

Describe the bug
The order of the qubit register changed after applying BigEndian or LittleEndian on it. The output of the following program is shown below. Is this an expected output when encoding a qubit register repeatedly?

q:[q:0,q:1,q:2]   q1:BigEndian([q:0,q:1,q:2])
q:[q:2,q:1,q:0]   q1:BigEndian([q:2,q:1,q:0])
q:[q:0,q:1,q:2]   q1:BigEndian([q:0,q:1,q:2])
q:[q:2,q:1,q:0]   q1:BigEndian([q:2,q:1,q:0])

To Reproduce
namespace NISLNameSpace {
	open Microsoft.Quantum.Intrinsic;
	open Microsoft.Quantum.Arithmetic;

	@EntryPoint()
	operation main() : Unit {
		for index in 0 .. 3{
			use q = Qubit[3];
			mutable q1 = BigEndian(q);
			Message($"q:{q}   q1:{q1}");
			ResetAll(q);
		}
	}
}

System information
operating system : Ubuntu 22.04 LTS
dotnet version : 6.0.400
QDK : 0.25.228311

This is not a change of order, but a result of qubit de-allocation. Qubits are de-allocated in reverse order and then pushed to a free-list. That's why when re-allocating q in loops with an odd index, the order will be reversed. You can print Message($"q:{q}"); before applying BigEndian to see that.

Thanks for your reply! It helps me a lot to understand the qubit allocation. What's more, I have a question when running the following programs. Why is the order same for q1 and q2? Dose it mean I use the wrong way to display the different encoding?
namespace NISLNameSpace {
	open Microsoft.Quantum.Intrinsic;
	open Microsoft.Quantum.Arithmetic;

	@EntryPoint()
	operation main() : Unit {
        use q = Qubit[3];
        Message($"q : {q}");
        mutable q1 = BigEndian(q);
        Message($"q1: {q1} {q1!}");
        mutable q2 = LittleEndian(q);
        Message($"q2: {q2} {q2!}");
        ResetAll(q);
	}
}

The output is:

q : [q:0,q:1,q:2]
q1: BigEndian([q:0,q:1,q:2]) [q:0,q:1,q:2]
q2: LittleEndian([q:0,q:1,q:2]) [q:0,q:1,q:2]

LittleEndian and BigEndian are not functions that change the array. You can better think of them as tags in this case. We create a type LittleEndian which is wrapping a qubit array but indicating that we interpret it in little-endian order, i.e., q[0] is the least-significant bit, whereas BigEndian indicates that q[0] is the most-significant bit. The algorithms need to ensure that this order is respected, but the type checker in the compiler can fail when I pass a LittleEndian type to an operation that expects BigEndian, even if they wrap the same qubit array.

Thanks a lot, I have fully understood these two types.


fix bug in get_coding

The get_coding function can return an encoding that is not valid. This caused spyder to crash during startup. This pull request makes sure only valid encodings are returned.
An example text which returns an invalid encoding:
# Try decoding as utf-8 bytestring        return __builtins__.unicode(msg, encoding="utf-8")    except:        pass    try:        # Try guessing system default encoding and decode as such        import locale        return __builtins__.unicode(msg, encoding=locale.getpreferredencoding())    except:        pass    try:        # Attempt using filesystem encoding        import sys        return __builtins__.unicode(msg, encoding=sys.getfilesystemencoding())    except:

The code looks fine but please replace c with a meaningful name like codec.

The variable name was changed. A working example:
import spyderlib
txt='return __builtins__.unicode(msg, encoding=sys.getfilesystemencoding())    except: '
coding = spyderlib.utils.encoding.get_coding(txt)
print(coding)

This returns a string which is not a valid coding.

Ok, changed code as suggested

@eendebakpt, thanks for your contribution!


array! macro from ndarray crate failing due to rustfmt adding trailing comma

When I have a macro call to the  array! macro from the ndarray crate, say:
let a = array![
            [1, 2, 3, 4],
            [5, 6, 7, 8],
            [9, 10, 11, 12],
            [13, 14, 15, 16]
        ]

Rustfmt will add a trailing comma to the last array part, turning it into:
let a = array![
            [1, 2, 3, 4],
            [5, 6, 7, 8],
            [9, 10, 11, 12],
            [13, 14, 15, 16],
        ]

Which results in a syntax error.
Is this fixable in a clean-ish way without too much special casing, or should I ask for a fix upstream to allow an optional trailing comma?

Does this reproduce in rustfmt-nightly latest version? And/or master? Because I think we have fixed this bug before.

I have the same problem:
$ rustfmt --version
0.9.0-nightly ( )

When I use macros from cpython:
    m.add(py,
             "save_data_as_xml",
             py_fn!(py,
                    save_data_as_xml_py(params: SomeStructName, out_name: &str, addon_param: &str)))?;

rustfmt format it to:
    m.add(
        py,
        "save_data_as_xml",
        py_fn!(
            py,
            save_data_as_xml_py(
                params: SomeStructName,
                out_name: &str,
                addon_param: &str,
            )
        ),
    )?;

The addon comma after addon_param: &str break compilation

@Dushistov can you also try with rustfmt-nightly (or master) please?

@nrc
I tried rustfmt-nightly (`0.2.1-nightly ( )):
the output little different, but additional comma in the same place, so build still FAILED:
    m.add(
        py,
        "save_data_as_xml",
        py_fn!(
            py,
            save_data_as_xml_py(params: SomeStructName, out_name: &str, addon_param: &str,)
        ),
    )?;

@nrc
Wait, I tryied with already modified sources by bad rustfmt,
will retry with clean sources

@nrc
With clean sources rustfmt-nightly (0.2.1-nightly ( )) works well:
-    m.add(py,
-             "save_data_as_xml",
-             py_fn!(py,
-                    save_data_as_xml_py(params: SomeStructName, out_name: &str, addon_param: &str)))?;
+    m.add(
+        py,
+        "save_data_as_xml",
+        py_fn!(
+            py,
+            save_data_as_xml_py(params: SomeStructName, out_name: &str, addon_param: &str)
+        ),
+    )?;

Confirmed fixed in nightly; only thing to do now is to wait for the next release. Closing.


OOTB Viva Connections Analytics

What type of issue is this?
Question
What SharePoint development model, framework, SDK or API is this about?
not applicable
Target SharePoint environment
SharePoint Online
What browser(s) / client(s) have you tested

[ ] 💥 Internet Explorer
[X] 💥 Microsoft Edge
[X] 💥 Google Chrome
[ ] 💥 FireFox
[ ] 💥 Safari
[ ] mobile (iOS/iPadOS)
[ ] mobile (Android)
[ ] not applicable
[ ] other (enter in the "Additional environment details" area below)

Additional environment details

browser version
SPFx version
Node.js version
etc

Issue description
Hi , Is there way to get  no of clicks (count) on out of the box Viva connections dashboard cards please .
Thanks.

@GowthamiKothapalli - sorry for the long delay.
Unfortunately, there is no way to get this kind of analytics.
But this is a good feedback.


isyncr v5.0.8

Current version of isyncr is v5.0.3 but v5.0.8 is now available. Can you please update to the latest?

Care to submit a PR? It isn’t difficult, and we’re here to help.

curl: (22) The requested URL returned error: 404 Not Found
Error: Download failed on Cask 'isyncr' with message: Download failed: http://www.jrtstudio.com/files/iSyncr 5.0.8.pkg


Upgrading LWJGL to version 3

Contains
LWJGL change:

Windowing system - GLFW
Input system - GLFW
GL methods, which use buffers (+suffix)
Some GL Extensions moved

We can replace jinput library to GLFW for joystics(controllers) after merge.
How to test
Checkout https://github.com/Terasology/CoreRendering/pull/15.
Run and play as usual, check :)
Platforms:

[ ] - MacOs
[ ] - Windows
[ ] - Linux(X)
[ ] - Linux(Wayland) - non goal, GLFW provides support for Wayland

Checklist:

Input system:

Keyboard:

keys working, keybindings working
chat and console receive chars as usual  - your language work, special symbols work, uppercase correct

Mouse:

position in NUI work correctly
scroll works
mouse acceleration  works correctly

Windowing system:

Windows resizing works
Fullscreen works
Pseudo fullscreen works

Rendering works, not glitches
OpenVR works (I hope)
Natives publishes in distribution
Check switchable graphics on MacOS

[bonus] measure performance on discrete graphics
Outstanding before merging

[x] Return SkySphere, LWJGL 3 lost glu packages
[x] Merge GLFW's char and key callbacks products... or separate NUIs key events
Its works separatelly now. Generates two independent KeyboardEvent streams.
[!] You are moving when type in chat or console in-game!
[x] Repair type adapter for configuration (Resolution not saves)
[ ] Repair LwjglPortlet (for TeraEd, or use another way)
[x] Repair framelimit
[x] Fix Resolution in settings gui (used Object's toString)
[x] Expose LWJGL 3 to modules
[x] Saving window position
[x] Refactor Debug/Error outputs for LWJGL 3
[x] Overall refactoring
[x] Replace SwingSplashScreen with LWJGL 3 solution (GLFW+ LWJGL's opengl).

Wow, thanks for putting all the effort into this @DarkWeird 👍 highly appreciated, I hope we can pull this change off 💪

this seems to work pretty well on my machine. mouse sensitivity seems to work a bit differently but other then that this works really well.

@pollend
Which os you use?

Distributor ID: Ubuntu
Description:    Ubuntu 20.04 LTS
Release:        20.04
Codename:       focal

Linux x64 enough :3

I tested on Linux (ubuntu 20.04, oracle jdk 11.0.7) and ran in to a few problems.
Breaks API
At least one API was broken which required patches to BasicCrafting and Behaviors in order to compile:
BasicCrafting
diff --git a/src/main/java/org/terasology/crafting/listCrafting/inHand/InHandCraftingButton.java b/src/main/java/org/terasology/crafting/listCrafting/inHand/InHandCraftingButton.java
index 5e131fd..88ffcc8 100644
--- a/src/main/java/org/terasology/crafting/listCrafting/inHand/InHandCraftingButton.java
+++ b/src/main/java/org/terasology/crafting/listCrafting/inHand/InHandCraftingButton.java
@@ -15,13 +15,13 @@
  */
 package org.terasology.crafting.listCrafting.inHand;

-import org.lwjgl.input.Keyboard;
 import org.terasology.input.BindButtonEvent;
 import org.terasology.input.DefaultBinding;
 import org.terasology.input.InputType;
+import org.terasology.input.Keyboard;
 import org.terasology.input.RegisterBindButton;

 @RegisterBindButton(id = "inHandCrafting", description = "In-hand crafting", category = "interaction")
-@DefaultBinding(type = InputType.KEY, id = Keyboard.KEY_F)
+@DefaultBinding(type = InputType.KEY, id = Keyboard.KeyId.F)
 public class InHandCraftingButton extends BindButtonEvent {
 }

Behaviors
diff --git a/src/main/java/org/terasology/grid/GridRendererButton.java b/src/main/java/org/terasology/grid/GridRendererButton.java
index 171e40a..56fa946 100644
--- a/src/main/java/org/terasology/grid/GridRendererButton.java
+++ b/src/main/java/org/terasology/grid/GridRendererButton.java
@@ -15,16 +15,16 @@
  */
 package org.terasology.grid;

-import org.lwjgl.input.Keyboard;
 import org.terasology.input.BindButtonEvent;
 import org.terasology.input.DefaultBinding;
 import org.terasology.input.InputType;
+import org.terasology.input.Keyboard;
 import org.terasology.input.RegisterBindButton;

 /**
  *
  */
 @RegisterBindButton(id = "grid_renderer", description = "Toggle grid renderer")
-@DefaultBinding(type = InputType.KEY, id = Keyboard.KEY_F6)
+@DefaultBinding(type = InputType.KEY, id = Keyboard.KeyId.F6)
 public class GridRendererButton extends BindButtonEvent {
 }

Since the class that got removed was inside lwjgl I guess it was incorrect for the modules to depend on it, but still some of modulespace is broken by this change.
Mouse Sensitivity
The mouse sensitivity was very low compared to develop (same config file for both). I had to turn it to max in the input menu and even then it felt sluggish. When switching back to develop I had to set it to the default value again.
Flaky Key Input
Key input was flaky in two cases:

Holding a movement key while moving fast (flight and hspeed active from console) resulted in those keys being "stuck" on sometimes. I.e. hold W until the game stutters to generate more chunks, then release W. Sometimes the character keeps moving even after W is released. Seems to miss a keyup event when stuttering.
Shift key in the console worked only occasionally. Holding shift then repeatedly pressing T resulted in only sporadic capitalization e.g. ttTtTTtttTttTttttTTtttttTTttttT. Doing the same against develop produced all capital T's as expected. Perhaps it's getting key repeat events?

Self-shaded chunks
Some chunks seemed to cast shadows on each other (not present when run against develop):
LWJGL3

Develop

SSAO Artifacts
Enabling SSAO led to strange artifacts around the player model:
SSAO on

SSAO off

@kaen
Thx for your work !

I looked in to this more deeply. There's a race condition because we're setting both of these callbacks:
        GLFW.glfwSetKeyCallback(window, this::glfwKeyCallback);
        GLFW.glfwSetCharCallback(window, this::glfwCharCallback);

GLFW doesn't define the order in which these fire, and both will fire for every keypress. The glfwCharCallback will try to consume any undispatched key events in the queue and replace them with the event it catches. I think this behavior explains both of the issues I found.

Good news! RU input and capitalization both worked great with these changes:

However, now typing in the console causes the character to move. Additionally, the console receives a tick mark character when I press that key to open it.
One more minor issue I found: during the splash screen and main menu load I get an "application is not responding" system popup (Ubuntu 20.04). The game still loads without issue but the pop shows over the application/splash screen and a less technical user might be thrown off by it:

Okay, tried on my Macbook with Java 10, that did not like this PR :-)
> Task :facades:PC:game FAILED
Recognized argument: -homedir
PathManager being initialized. Initial code location is /Users/cervator/dev/terasology/git_ws/mini/Terasology/engine/build/classes
Switched it to expected working dir: /Users/cervator/dev/terasology/git_ws/mini/Terasology
installPath was null, running from IDE or headless server? Setting to: /Users/cervator/dev/terasology/git_ws/mini/Terasology
Found the natives dir: /Users/cervator/dev/terasology/git_ws/mini/Terasology/natives
2020-07-09 23:43:04.742 java[6239:139521] WARNING: nextEventMatchingMask should only be called from the Main Thread! This will throw an exception in the future.
2020-07-09 23:43:04.742 java[6239:139521] *** Assertion failure in +[NSUndoManager _endTopLevelGroupings], /BuildRoot/Library/Caches/com.apple.xbs/Sources/Foundation/Foundation-1575.19/Foundation/Misc.subproj/NSUndoManager.m:361
2020-07-09 23:43:04.742 java[6239:139521] *** Assertion failure in +[NSUndoManager _endTopLevelGroupings], /BuildRoot/Library/Caches/com.apple.xbs/Sources/Foundation/Foundation-1575.19/Foundation/Misc.subproj/NSUndoManager.m:361
2020-07-09 23:43:04.742 java[6239:139521] *** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: '+[NSUndoManager(NSInternal) _endTopLevelGroupings] is only safe to invoke on the main thread.'
*** First throw call stack:
(
        0   CoreFoundation                      0x00007fff3d6112fd __exceptionPreprocess + 256
        1   libobjc.A.dylib                     0x00007fff67ce2a17 objc_exception_throw + 48
        2   CoreFoundation                      0x00007fff3d62c016 +[NSException raise:format:arguments:] + 98
        3   Foundation                          0x00007fff3f8cb791 -[NSAssertionHandler handleFailureInMethod:object:file:lineNumber:description:] + 194
        4   Foundation                          0x00007fff3f7fe80e +[NSUndoManager(NSPrivate) _endTopLevelGroupings] + 473
        5   AppKit                              0x00007fff3ab4a681 -[NSApplication run] + 916
        6   libglfw.dylib                       0x0000000133f2ca85 libglfw.dylib + 68229
        7   libglfw.dylib                       0x0000000133f26416 libglfw.dylib + 42006
        8   ???                                 0x000000011a6a1b50 0x0 +<PHONE_NUMBER>
)
libc++abi.dylib: terminating with uncaught exception of type NSException

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':facades:PC:game'.
> Process 'command '/Library/Java/JavaVirtualMachines/jdk-10.0.1.jdk/Contents/Home/bin/java'' finished with non-zero exit value 134

Tried separately running ./gradlew extractNatives just in case something hadn't updated, but that didn't help :-(

@kaen ok. Seems, it is because i remove GLFW.pollEvents for splashscreen.
@Cervator ah, splashscreen use separate thread... it must work.. I will investigate it.

@Cervator
https://github.com/libgdx/libgdx/pull/4472
set jvmargs  to "-XstartOnFirstThread" for macos run, plz.
it can help... or not help and then i will need rething splashscreen running :(
MacOS specific problem.

All GLFW-LWJGL code moved to https://github.com/MovingBlocks/SplashScreen
See https://github.com/MovingBlocks/SplashScreen/pull/4

Did some limited testing on Mac, resolving the minor conflicts and changing the splash-screen version goal to 1.1.0-SNAPSHOT just until we can finalize that piece. Noted a compile issue with Behaviors (just in a smaller workspace, not every module present) - something about Keyboard
Ran the shiny new "TerasologyPC for Mac" which includes the -XstartOnFirstThread but it still failed :-(

Dangit. I tried to commit my merge to get this branch up to date with conflicts resolved and the little adjustment to the splash screen version. But that brought in the new commits as if they're unique changes, not clean merges, as far as I can tell. This is an example of a Git concern I raised on Discord a little while ago, although I'm still not sure how I'm provoking it and if it is just me doing something bad.
This badly needs a rebase with squash anyway by the time we actually want to merge it, so I guess that'll clean up the history. Sorry about making your branch muddy, @DarkWeird !

I made some tweaks to mouse to correct for some stuttering issues I've been having:
https://github.com/DarkWeird/Terasology/pull/1

 LWJGL3

Did final testing on both Mac + Windows, made a few tweaks via #4216, and merged this. Thanks for both the momentous work and the patience in keeping it up to date and continuously improving it @DarkWeird :-)


cURL error 7: (see https://curl.haxx.se/libcurl/c/libcurl-errors.html)

i hit this error when login. May i know what is this related?

It looks like the error isn’t complete. Can you paste the error in a comment? Did you modify the config file to connect to your VBO365 server correctly?

cURL error 7: (see https://curl.haxx.se/libcurl/c/libcurl-errors.html)
the error is get when i login and have set the config.

[root@localhost ~]# cat /var/www/html/config.php

================================================

Error 7 means it couldn't connect, could it be you have a firewall running on the VBO365 server (make sure port 4443 is open.


Xarango id is in _key not id

This will grab the _key (which is id in arangodb) and use it as the id.
_id in arangodb is collection/key

@jnylen thanks for the PR but I don't think we want to include this change in the core library.
If we try to handle all possible cases for id/1 we'll forever be adding new function signatures.  To address this, we allow you to override the id/1 function to handle things specific to your application like this!
https://github.com/jnylen/jsonapi/blob/e161ef927abd53d2f116229dc1a8e98c40951a58/lib/jsonapi/view.ex#L202
Happy to answer any questions you have about implementing this in your project 👍

@doomspork,
Oh, I totally missed that variant. I'll try that instead. Thanks for a great library!


Pack validation changes

This pull request updates pack metadata file so they will pass validation with new, to be released version of StackStorm.
Background
Latest version of StackStorm changes some pack validation rules (version string now needs to be a valid semver identifier, email field needs to contain valid email address, etc.) and the pack.yaml is now also correctly validated the registration phase (previously there was a bug in the code so the pack metadata file wasn't actually validated).
TODO

[ ] Add ref attribute with a value which matches directory name to each pack.yaml

I also confirmed this doesn't break current stable and older versions.


Microphone Control

With regards to the doc, https://docs.microsoft.com/en-us/powerapps/maker/canvas-apps/controls/control-microphone
Is there a Start and Stop recording based on the User's Pause or start speaking, without touching the buttons?  Can Microphone sense if someone speaking or stop speaking? Thanks.

Hi there, unfortunately no.  There is no way to detect when a recording should start / end.  We also don't have this feature on our radar in the short term.  Sorry for any inconvenience.

Thanks for answering.


GPS Accuracy

Hi;
How can i get gps accuracy?

Hi,
can you please elaborate on your question?
to get the GPS accuracy you can use getaccuracy method on location object.


range_cache_memeory_engine: fix panic when pending region split before start batch loading

What is changed and how it works?

Issue Number: ref #16141

What's Changed:
Fix the bug that pending region split before batch loading start can
cause panic.

When handle LoadRegion event, the background worker only update the
region with the same id to Loading state, but the region is possible
already split so other region within the same range will still in
`ReadyToLoad` state. So it will cause panic in
`on_snapshot_load_finished`.

The PR removes the useless intermediate state `ReadyToLoad`. After this
change, the raft apply worker will directly update state from Pending
to Loading before scheduler LoadRegion task. So the background worker
will not try to update target region's state.

Related changes

[ ] PR to update pingcap/docs/pingcap/docs-cn:
[ ] Need to cherry-pick to the release branch

Check List
Tests

[x] Unit test
[ ] Integration test
[ ] Manual test (add detailed scripts or steps below)
[ ] No code

Side effects

[ ] Performance regression: Consumes more CPU
[ ] Performance regression: Consumes more Memory
[ ] Breaking backward compatibility

Release note

None

By the way, please file a bug issue.


Test issue

To test the GitHub action of managing issues.

The test failed, closing this issue now.


Rebuild rows after breakpoint

Add breakpoint rebuild row when number of rows is one or more

@kenwheeler lookup this

+1

+1

+1

+11 ! oneoneone! eleveneleven!

+1!one!sin(90)

+111

:fireworks:

can you add tag? plz

@emgiezet +1

Tag?

up version to bower repo

Reverted, this broke master

If rows === 1, why do you need to rebuild them. they shouldn't be there at all?

Can you show me an example in a jsfiddle of what this is supposed to fix?


Cannot send cart from OPAC in 22.11

GetMarcAuthors has been removed, see the reference Koha bug below:
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27266

Hello!  We've just issues a PR to resolve this issue in 23.11.  Please feel free to test it here: #138

#138 fixes carts in 23.11.


[ISSUE #1000]✨Add RocketMQTokioRwLock and RocketMQTokioMutex for rocketmq crate🎨

Which Issue(s) This PR Fixes(Closes)

Fixes #1000
Brief Description

How Did You Test This Change?

Summary by CodeRabbit

New Features

Introduced a new module for asynchronous locking mechanisms: rocketmq_tokio_lock.
Added RocketMQTokioMutex for exclusive data access.
Added RocketMQTokioRwLock for concurrent read and write access, with timeout options.

These enhancements improve the library's capabilities for asynchronous programming, providing users with more efficient synchronization tools.

🚀Thanks for your contribution🎉. CodeRabbit(AI) will review your code first🔥

🚀Thanks for your contribution🎉. CodeRabbit(AI) will review your code first🔥

🚀Thanks for your contribution🎉. CodeRabbit(AI) will review your code first🔥

🚀Thanks for your contribution🎉. CodeRabbit(AI) will review your code first🔥

🚀Thanks for your contribution🎉. CodeRabbit(AI) will review your code first🔥

🚀Thanks for your contribution🎉. CodeRabbit(AI) will review your code first🔥

🚀Thanks for your contribution🎉. CodeRabbit(AI) will review your code first🔥

🚀Thanks for your contribution🎉. CodeRabbit(AI) will review your code first🔥

🚀Thanks for your contribution🎉. CodeRabbit(AI) will review your code first🔥


Clarifying target audience

After a conversation with a new-comer, it seems that it wasn't clear who is allowed to join. Trying to clarify this.

Makes a lot of sense! Thank you for adding this


Update production database to PostgreSQL 16

The current production specprod database is based on PostgreSQL 11, which no longer supported. I have already created a Spin image based on PostgreSQL 16 and tested it on the Spin development cluster.  The Dockerfile and related files are included in PR #14.
To avoid extended downtime on the existing production database, set up a separate database instance with persistent storage, load fuji, guadalupe and iron from pg_dump files, then proceed to load daily & kibo.

It might also be a good idea to re-create the pg_dump files. I recall there may have been minor changes applied to fuji, guadalupe and iron after they were loaded.

As of today:

A new PostgreSQL 16 container has been created; it's called db2.
New pg_dump files for fuji, guadalupe and iron have been created and backed up to HPSS.
fuji, guadalupe and iron have been loaded into db2.

And remaining to do:

The intention is that db2 will be used to load kibo and daily.
After kibo is loaded and tile-based loading has been tested for daily, db2 will be renamed to db, at which point it will take over as the official specprod-db server.
Any old resources used by the previous PostgreSQL 11 container will be released.

In the end it was simpler to redirect specprod-db.desi.lbl.gov to point to db2. The original db instance can now be removed when convenient.


add image pipeline TF

Pulling pipelines into different tf modules to allow for spinning up different versions of it for the forked TEI work.
Something like:

Do you not need to remove the corresponding TF files from the root of the stack module? e.g. https://github.com/wellcomecollection/catalogue-pipeline/blob/feat/image_pipelines_tf/pipeline/terraform/stack/service_image_inferrer.tf
Otherwise won't you create two image inferrers and two image ingestors?

This is also very wip while I'm on leave.

Ah cool, for some reason I thought you'd asked for a review on this. Ignore me! 😅

Closing in favour of @alicefuzier multiple PRs on this.


Get the HDMI2EthernetSoc running at 1Gbps

From @enjoy-digital

You can use MiniSoC with 1Gps Ethernet (but since data are generated by the CPU, you will not have 1Gbps), but the HDMI2EthernetSoc only supports 100Mbps for now (since sys_clk is < to 125MHz).

@aomtoku - Please take a look at this github issue.

This is working now it seems.


1274 test user cascade delete

@mjstealey Handles issue #1274. Answer is yes, deleting a user deletes the access control information for that user.
User.delete() correctly cascades to deleting UserAccess records in access control. This test proves that.
The fact that UserAccess deletion cascades to delete UserAccess records from Groups and Resources is tested elsewhere.

@pkdash I am submitting this without the resource preservation tests, which fail for a yet unknown reason.
For now, this will work, because the intent of this is to delete users that have not yet completed registration and thus can't own resources.

@alvacouch Since the user profile page allows a user to de-activate his/her account, there is a possibility of inactive users owning resources.

@alvacouch, @pkdash - Note, this PR is not failing due to unit tests, but due to the new flake8 check that is in place. It is presently set to 0 warnings = pass, 1+ warnings = fail
Overview: http://ci.hydroshare.org:8080/job/github-pr-builder-hydroshare/134/
Specifics: http://ci.hydroshare.org:8080/job/github-pr-builder-hydroshare/134/artifact/pylint.out/view/

@mjstealey @pkdash Will fix. No problem. However, I have some trepidation about the current behavior. I am still thinking about how to resolve that.

Problem is actually here; [https://github.com/hydroshare/hydroshare/blob/develop/hs_core/models.py#L72]
The default is null=False and on_delete=models.CASCADE.
Thus removing the creator removes the resource.

Note: it is due to this that -- in Django 2.0 -- on_delete becomes a required element in all foreign key fields. So you must be explicit about whether delete cascades or not.

@pkdash @mjstealey If we don't want this behavior, the best practice is on_delete=models.RESTRICT, which would prevent the user from being deleted.

@mjstealey @pkdash Looks like the default is always CASCADE, even if null=False. Thus last_changed_by also creates the behavior.
Also, although the SQL syntax is RESTRICT, the proper Django syntax is
on_delete=models.PROTECT
This would have to be set for all User foreign keys for BaseResource in order to avoid the anomaly.

@alvacouch Thanks for this update. This seems a like small change to fix the problem. But as it relates to deleting inactive users, we should only delete those users who never activated their accounts.

@alvacouch Not sure if you are planning to do any more work on this PR. Otherwise I can +1

+1 from me then.


Reimplement capture toolbar in Qt rather than ImGui.

The toolbar now auto-resizes based on the window width. Fixes b/158244053.
Note that I removed the process name.  Not sure it adds tremendous value and it takes a lot of screen real estate. We could always have it in the title bar similar to what we see when we load a capture.

Just from the screenshot: What about taking it out of the "capture" tab and positioning it directly under the menu bar like toolbars normally are? This way it would always use the full width of the window (I don't think it's a problem if we lose a tiny bit of vertical space on the right side). Otherwise I have a feeling that when the capture tab is not that wide it would take several lines. Also some buttons don't only apply to the capture view, for example load, save, feedback.

I like Davide's idea of using the designer for creating the toolbar.
This would probably require to move the icon-loading-code to Qt's qrc system for seeing and setting the icons in the designer.

I really want to minimize vertical space taken up by UI.

Is vertical space more important in the capture view or in the right-hand-side tabs? How does the toolbar scale when it doesn't fit in width in the capture view? If it ends up taking new lines, then more vertical space is taken from the capture view.

I didn't think it made sense for all the actions to be available globally.

having the toolbar at the top level would expose functionality that is not at all pertinent to any other tabs than the capture view (start/stop capture without visual feedback is a risk; filter tracks and the timer label wouldn't make sense either in any other view.

Of course. Some actions would have to be hidden based on whether the "capture" tab is currently selected, but some others (like load and save—honestly for feedback there's a dedicated label a two centimetres above a so that button for me could just go) should always be available.
Also start and stop capture would still provide a feedback as the buttons would at least change state and the timer would start/stop. If it was possible to hit the start button while in the "home" tab and have that switch automatically to the "capture" tab, that would even save users from having to always click on the "capture" tab first.

Some of those actions are already present in the main menu, so they are accessible from anywhere.

Based on this reasoning some actions should just not be on the toolbar at all.
Bottom line of what I'm saying it that having a toolbar with global actions and actions that even if not global are still main actions of Orbit and might be useful regardless of what tab is selected (capturing) as part of a tab is weird. Normal positioning of a toolbar in basically all desktop applications is right below the menu taking all the width (which also mean that it scales only with the window, not with the window and the left pane).

How does the toolbar scale when it doesn't fit in width in the capture view?

It doesn't take new lines, toolbar elements disappear "gracefully" unlike the previous implementation.

Bottom line of what I'm saying is that having a toolbar with global actions and actions that even if not global are still main actions of Orbit and might be useful regardless of what tab is selected (capturing) as part of a tab is weird.

Maybe we should get input from UX here.  I would suggest having the start capture action globally mapped to F5 and automatically focusing on the capture tab when pressed. Stop capture would be mapped to Shift-F5. We'd still have the buttons in the capture view.  Maybe we should have a "Capture" menu as well. I still don't see the value of having a global toolbar.

Normal positioning of a toolbar in basically all desktop applications is right below the menu taking all the width

I disagree. Firstly because I really think it's a useless waste of space, but also because modern applications such as Visual Studio Code for example are moving away from this pattern by having toolbars that are contextual, take up minimal space, and can definitely be per-tab.
If we really wanted to have a global toolbar, we'd still want to have the capture specific toolbar, something like this:

I'd suggest moving this conversation to another channel and submitting the change for now as it is an improvement over the previous implementation and is basically a 1:1 mapping in functionality.

I still have a lot of thoughts about your reply but we should take it offline as it would probably be mostly repeating what I said before and nobody will change his opinion, so I'll focus on the main point.

It doesn't take new lines, toolbar elements disappear "gracefully" unlike the previous implementation.

it is an improvement over the previous implementation and is basically a 1:1 mapping in functionality.

Toolbar elements shouldn't just disappear when the width of the Window is much more (usually double) than the space available for the toolbar, and this is the main problem that should be solved, not how graceful they disappear, that's why it shouldn't be a 1:1 mapping.
The suggestion of having a single global toolbar with some controls appearing when the "capture" tab is selected killed two birds with one stone: it allowed more space for the toolbar and provided basic global actions (including the very main action of the application) always one click away, regardless of the current tab.

Not touching on points that are in the discussion, but pragmatically I would say that this PR is an improvement over the existing implementation. Even if we can't agree right now on how to do the design of the toolbar (or multiple toolbars), this PR should be merged and we can then iterate on the design.


Looks like an amazing project!

How does it compare to the good-old, published-properties-only RTTI, and the new Extended RTTI in Delphi?
I guess it's something in between? That means, more feature rich than the traditional RTTI, but faster than the new Extended RTTI? I'm just guessing :)

Hello :)
The project has the following conceptual features:

Standard types, including FreePascal, old and new versions of Delphi, convenient functions for managing them.
Universal compiler and programming language data representation.
Convert standard RTTI to universal data representation.
Minimizing dependencies on heavy units like Classes, Variants, Generics, etc.
Cross-platform functions invoke, including FreePascal and old versions of Delphi, creation of function and interface interpreters.
Marshalling (serialization and deserialization of data) through different formatters: JSON, XML, FlexBin and others.
Easy to use unit testing library that does not access the memory manager.

@d-mozulyov  Thanks for the details. What's the performance when comparing with the built-in RTTI?

In what cases are you interested in high performance?

For example, I enjoyed using DuckDuckDelphi a lot but it uses Extended RTTI, and I had to abandoned it in some cases such as looping through an object list coz the performance is really really bad...

I specialize in optimizations. But I'm not sure that the library will help in your case, because I do not know your case. For me personally, the speed of serialization is important. I developed the binary FlexBin format, here you can see the preliminary results of the formatter for this format. The formatter has not yet been published, but over time it will become part of the library.

Thanks for your explanation, I'm sure you are specialized in optimization :+1:
What's the intended use of this library, it's serialization and deserialization, right?

I have been working with RTTI for a long time and at one moment noticed that I lacked a universal solution. Therefore, I transfer part of the developments to the library and gradually expand them.
I have the following needs: scripti binding, serialization and testing.

@edwinyzh I saw mentioned before that DuckDuckDelphi suffers from quite some overhead from enhanced RTTI - however working a lot with RTTI in my libraries I know that this is not often the case because certain operations that can easily be cached are done over and over again and thus are an issue in the library that used RTTI.
One example is the base class that automatically initializes its field depending on some attributes you put on them including creating classes for TStrings fields and alike - some third party library had this feature but did that with like half a dozen abstraction layers upon RTTI which turned this into some performance nightmare while I implemented a similar thing in Spring.TManagedObject which has a performance that is almost as fast as compiled code (because rtti is only read once and then cached in some highly optimized data structures)

@sglienke thank you for your valuable comments, caching is a proven technique to improvement performance, I believe. But according to what I've read, such as the blog by "ab" the author of the mORMot framework, the new Extended RTTI actually slower than the old "published" RTTI, but I'd love to see more source and benchmarks if you know some.
That being said, If I would have some time and want to use duckduckDelphi again, can I'll see what I can improve the performance of it with the caching technique you mentioned, thank you very much for the hint.
PS, by "reading the rtti info at once", does spring4D do that at program startup? I find it will cause a near-one-second startup delay for a project that uses spring4d collections. spring4d is an excellent, but just like Generics.Collections, I kinda think the Delphi code editor/code completion/code insight doesn't match it due to the lack of support for generics and annon. methods...
PS2, Out of curiosity what the reason you have chosen bitbucket for hosting spring4D to begin with? I found it with github it's easier to communicate with each.

I don't want to hijack this for discussing other matters - if you have questions regarding spring4d please ask them on our forum (link is in the readme.md) or contact me directly.

You are definitely a nice Internet communicator :)

@edwinyzh
Please tell us what are your needs for RTTI and performance?

OK, to sum it up, the reason why I found my interest in this project is that, I enjoyed using duckduckdelphi, because it's a great time-saver that allowed me to operate on a form like the following:
Form1.duck.all.has('visible').sett(false);
or
duck.all.can('Clear').go;
And although code like won't suffer performance issue, it doesn't perform well with things like TList<TObject>, and without looking at the code I wrongly concluded the performance issue it has came from the Extended RTTI feature of Delphi, and thought your Tiny.Rtti project might be able to solve the issue of it. That's where my interests came from.
But as you see above, @sglienke said the issue can be solved using caching, and I trust him.

@edwinyzh
It would be great if you started developing the Tiny.DuckTyping unit.  But other factors also affect performance. Searching for an identifier in a table, executing a function, changing a property. All this is planned to be accelerated as much as possible within the framework of Tiny.Rtti, Tiny.Invoke and Tiny.Properties units.

@d-mozulyov ,
Tiny.RTTI has a great potential, but I'm not in a position to allowed me start that kinda of project yet :)

@edwinyzh
No problem. Stay tuned for updates (the next release will be in a few days). As soon as the opportunity arises, you can start developing Tiny.DuckTyping.


Simple count mv sanity check failure

Describe the bug
dev=> explain create materialized view v as select count(*) c from sbtest1;
                           QUERY PLAN
----------------------------------------------------------------
 StreamMaterialize { columns: [c], pk_columns: [] }
 └─StreamProject { exprs: [sum0(count)] }
   └─StreamGlobalSimpleAgg { aggs: [sum0(count), sum0(count)] }
     └─StreamExchange { dist: Single }
       └─StreamStatelessLocalSimpleAgg { aggs: [count, count] }
         └─StreamTableScan { table: sbtest1, columns: [id] }

thread 'risingwave-streaming-actor' panicked at 'inconsistent update!
table_id: 1005, vnode: 0, key: Row([])
stored value: Some(Row([Some(Int64(80780)), Some(Int64(80780))]))
expected value: Row([Some(Int64(205200)), Some(Int64(205200))])
new value: Row([Some(Int64(206224)), Some(Int64(206224))])', /Users/dylan/Desktop/playground/risingwave/src/storage/src/table/streaming_table/state_table.rs:735:13
stack backtrace:
   0: rust_begin_unwind
             at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/panicking.rs:575:5
   1: core::panicking::panic_fmt
             at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/core/src/panicking.rs:65:14
   2: risingwave_storage::table::streaming_table::state_table::StateTable<S>::do_update_sanity_check::{{closure}}
             at ./src/storage/src/table/streaming_table/state_table.rs:735:13
   3: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/core/src/future/mod.rs:91:19
   4: risingwave_storage::table::streaming_table::state_table::StateTable<S>::batch_write_rows::{{closure}}
             at ./src/storage/src/table/streaming_table/state_table.rs:640:29
   5: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/core/src/future/mod.rs:91:19
   6: risingwave_storage::table::streaming_table::state_table::StateTable<S>::commit::{{closure}}
             at ./src/storage/src/table/streaming_table/state_table.rs:589:57
   7: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/core/src/future/mod.rs:91:19
   8: risingwave_stream::executor::global_simple_agg::GlobalSimpleAggExecutor<S>::flush_data::{{closure}}
             at ./src/stream/src/executor/global_simple_agg.rs:251:39
   9: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/core/src/future/mod.rs:91:19
  10: risingwave_stream::executor::global_simple_agg::GlobalSimpleAggExecutor<S>::execute_inner::{{closure}}
             at ./src/stream/src/executor/global_simple_agg.rs:332:22
  11: <futures_async_stream::try_stream::from_generator::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-async-stream-0.2.5/src/lib.rs:430:31
  12: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.24/src/stream.rs:120:9
  13: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.24/src/stream.rs:120:9
  14: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.24/src/stream/stream/mod.rs:1626:9
  15: <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.24/src/stream/stream/next.rs:32:9
  16: <minitrace::future::InSpan<T> as core::future::future::Future>::poll
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/minitrace-0.4.0/src/future.rs:118:19
  17: risingwave_stream::executor::wrapper::trace::trace::{{closure}}
             at ./src/stream/src/executor/wrapper/trace.rs:53:60
  18: <futures_async_stream::try_stream::from_generator::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-async-stream-0.2.5/src/lib.rs:430:31
  19: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.24/src/stream.rs:120:9
  20: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.24/src/stream/stream/mod.rs:1626:9
  21: <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.24/src/stream/stream/next.rs:32:9
  22: <async_stack_trace::StackTraced<F,_> as core::future::future::Future>::poll
             at ./src/utils/async_stack_trace/src/lib.rs:178:23
  23: risingwave_stream::executor::wrapper::trace::stack_trace::{{closure}}
             at ./src/stream/src/executor/wrapper/trace.rs:120:70
  24: <futures_async_stream::try_stream::from_generator::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-async-stream-0.2.5/src/lib.rs:430:31
  25: risingwave_stream::executor::wrapper::schema_check::schema_check::{{closure}}
             at ./src/stream/src/executor/wrapper/schema_check.rs:24:1
  26: <futures_async_stream::try_stream::from_generator::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-async-stream-0.2.5/src/lib.rs:430:31
  27: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.24/src/stream.rs:120:9
  28: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.24/src/stream/stream/mod.rs:1626:9
  29: <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.24/src/stream/stream/next.rs:32:9
  30: risingwave_stream::executor::wrapper::epoch_check::epoch_check::{{closure}}
             at ./src/stream/src/executor/wrapper/epoch_check.rs:31:44
  31: <futures_async_stream::try_stream::from_generator::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-async-stream-0.2.5/src/lib.rs:430:31
  32: risingwave_stream::executor::wrapper::update_check::update_check::{{closure}}
             at ./src/stream/src/executor/wrapper/update_check.rs:27:1
  33: <futures_async_stream::try_stream::from_generator::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-async-stream-0.2.5/src/lib.rs:430:31
  34: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.24/src/stream.rs:120:9
  35: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.24/src/stream.rs:120:9
  36: risingwave_stream::executor::simple::SimpleExecutorWrapper<E>::execute_inner::{{closure}}
             at ./src/stream/src/executor/simple.rs:68:5
  37: <futures_async_stream::try_stream::from_generator::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-async-stream-0.2.5/src/lib.rs:430:31
  38: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.24/src/stream.rs:120:9
  39: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.24/src/stream.rs:120:9
  40: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.24/src/stream/stream/mod.rs:1626:9
  41: <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.24/src/stream/stream/next.rs:32:9
  42: <minitrace::future::InSpan<T> as core::future::future::Future>::poll
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/minitrace-0.4.0/src/future.rs:118:19
  43: risingwave_stream::executor::wrapper::trace::trace::{{closure}}
             at ./src/stream/src/executor/wrapper/trace.rs:53:60
  44: <futures_async_stream::try_stream::from_generator::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-async-stream-0.2.5/src/lib.rs:430:31
  45: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.24/src/stream.rs:120:9
  46: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.24/src/stream/stream/mod.rs:1626:9
  47: <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.24/src/stream/stream/next.rs:32:9
  48: <async_stack_trace::StackTraced<F,_> as core::future::future::Future>::poll
             at ./src/utils/async_stack_trace/src/lib.rs:178:23
  49: risingwave_stream::executor::wrapper::trace::stack_trace::{{closure}}
             at ./src/stream/src/executor/wrapper/trace.rs:120:70
  50: <futures_async_stream::try_stream::from_generator::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-async-stream-0.2.5/src/lib.rs:430:31
  51: risingwave_stream::executor::wrapper::schema_check::schema_check::{{closure}}
             at ./src/stream/src/executor/wrapper/schema_check.rs:24:1
  52: <futures_async_stream::try_stream::from_generator::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-async-stream-0.2.5/src/lib.rs:430:31
  53: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.24/src/stream.rs:120:9
  54: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.24/src/stream/stream/mod.rs:1626:9
  55: <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.24/src/stream/stream/next.rs:32:9
  56: risingwave_stream::executor::wrapper::epoch_check::epoch_check::{{closure}}
             at ./src/stream/src/executor/wrapper/epoch_check.rs:31:44
  57: <futures_async_stream::try_stream::from_generator::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-async-stream-0.2.5/src/lib.rs:430:31
  58: risingwave_stream::executor::wrapper::update_check::update_check::{{closure}}
             at ./src/stream/src/executor/wrapper/update_check.rs:27:1
  59: <futures_async_stream::try_stream::from_generator::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-async-stream-0.2.5/src/lib.rs:430:31
  60: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.24/src/stream.rs:120:9
  61: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.24/src/stream.rs:120:9
  62: risingwave_stream::executor::mview::materialize::MaterializeExecutor<S>::execute_inner::{{closure}}
             at ./src/stream/src/executor/mview/materialize.rs:125:5
  63: <futures_async_stream::try_stream::from_generator::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-async-stream-0.2.5/src/lib.rs:430:31
  64: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.24/src/stream.rs:120:9
  65: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.24/src/stream.rs:120:9
  66: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.24/src/stream/stream/mod.rs:1626:9
  67: <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.24/src/stream/stream/next.rs:32:9
  68: <minitrace::future::InSpan<T> as core::future::future::Future>::poll
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/minitrace-0.4.0/src/future.rs:118:19
  69: risingwave_stream::executor::wrapper::trace::trace::{{closure}}
             at ./src/stream/src/executor/wrapper/trace.rs:53:60
  70: <futures_async_stream::try_stream::from_generator::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-async-stream-0.2.5/src/lib.rs:430:31
  71: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.24/src/stream.rs:120:9
  72: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.24/src/stream/stream/mod.rs:1626:9
  73: <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.24/src/stream/stream/next.rs:32:9
  74: <async_stack_trace::StackTraced<F,_> as core::future::future::Future>::poll
             at ./src/utils/async_stack_trace/src/lib.rs:178:23
  75: risingwave_stream::executor::wrapper::trace::stack_trace::{{closure}}
             at ./src/stream/src/executor/wrapper/trace.rs:120:70
  76: <futures_async_stream::try_stream::from_generator::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-async-stream-0.2.5/src/lib.rs:430:31
  77: risingwave_stream::executor::wrapper::schema_check::schema_check::{{closure}}
             at ./src/stream/src/executor/wrapper/schema_check.rs:24:1
  78: <futures_async_stream::try_stream::from_generator::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-async-stream-0.2.5/src/lib.rs:430:31
  79: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.24/src/stream.rs:120:9
  80: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.24/src/stream/stream/mod.rs:1626:9
  81: <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.24/src/stream/stream/next.rs:32:9
  82: risingwave_stream::executor::wrapper::epoch_check::epoch_check::{{closure}}
             at ./src/stream/src/executor/wrapper/epoch_check.rs:31:44
  83: <futures_async_stream::try_stream::from_generator::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-async-stream-0.2.5/src/lib.rs:430:31
  84: risingwave_stream::executor::wrapper::update_check::update_check::{{closure}}
             at ./src/stream/src/executor/wrapper/update_check.rs:27:1
  85: <futures_async_stream::try_stream::from_generator::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-async-stream-0.2.5/src/lib.rs:430:31
  86: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.24/src/stream.rs:120:9
  87: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /Users/dylan/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.24/src/stream.rs:120:9
  88: <risingwave_stream::executor::dispatch::DispatchExecutor as risingwave_stream::executor::StreamConsumer>::execute::{{closure}}
             at ./src/stream/src/executor/dispatch.rs:260:9
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

*** async stack trace context of current task ***

Actor 25: `SELECT count(*) AS c FROM sbtest1` [13.335491291s]
  Epoch<PHONE_NUMBER>413632 [447.514875ms]
    MaterializeExecutor<PHONE_NUMBER> (actor 25, executor 67) [447.505375ms]
      ProjectExecutor<PHONE_NUMBER> (actor 25, executor 66) [447.517749ms]
        GlobalSimpleAggExecutor-1900000040 (actor 25, executor 64) [447.5155ms]  <== current

To Reproduce

Change risingwave.toml

barrier_interval_ms = 10
checkpoint_frequency = 2

Use sysbench to keep inserting data to table sbtest1

sysbench \
--db-driver=pgsql \
--pgsql-host=<IP_ADDRESS>  \
--pgsql-port=4566  \
--pgsql-user=root \
--pgsql-db=dev \
--threads=16 \
--report-interval=1 \
--events=0 \
--time=10 \
--percentile=99 \
--auto_inc=false \
--table_size=100000000 \
--db-ps-mode=disable \
--skip_trx=true \
--range_size=1 \
--random_points=1 \
/usr/local/share/sysbench/select_random_points.lua \
prepare

Create a mv (with another terminal so that we can run create mv concurrently with the insert statements generated by the sysbench)

 create materialized view v as select count(*) c from sbtest1;

Wait a minute and then it will fail.

Expected behavior
No response
Additional context
No response

Findings:

There is indeed a data loss in reading the simple agg state via get
Right before the data loss happens, there is a L0 -> L0 sub-level compaction triggered

Chatted with @Little-Wallace offline. We suspect this is due to the recent merged delete range support breaking some previous assumptions on SSTs.

Yes, I believe so. I have logged all the update of GlobalSimpleAggExecutor and find that at some point the executor read the state with latest epoch but read some old data far from now.

@Little-Wallace found out that this bug is introduced by #5998. Previously we assume for a given user key cannot exists in more than one SSTs in a given level and we use user key to do binary search during get.
After #5998, compactor uses full key instead of user key to find out overlapping SSTs to construct a compaction task to take delete range tombstone into consideration. However, this will break the assumption mentioned above and thus causing incorrect result in get. We plan to revert this change first in #6364 to fix the issue since range deletion is not used and refactoring on the range deletion side will be done in a separate PR.


Date of Birth theme

Hi I need a quick way for user to select day, month year. instead of clicking heaps of times to get to the year.
Is there anyway to make it show year first the months and then day?

Use the viewMode option to 'years'