rename group id to upload "Maven Central Repository"

@koijima mod group id for https://issues.sonatype.org/browse/OSSRH-33140.
I changed the config package groupId on yahoojapan/dataplatform-config/pull/2 and schema lib https://github.com/yahoojapan/dataplatform-schema-lib/pull/2

Thx


Embedding setup tool only creates traditional workspace and is not compatible with ServicePrincipal authentication

This article states "if you use service principal to sign into your application, then you're required to you use the new workspaces"
Where this is the case, readers should be warned NOT to use the 'Embedding setup tool'.
I wasted a day doing this and was unable to get the resulting 'App Owns Data' sample app working because the automated tool creates the wrong sort of workspace (traditional), which is not compatible with ServicePrincipal authentication type (this is because you cannot make the ServicePrincipal an Admin of a traditional workspace).
I was able to work around this by manually creating a 'new' workspace.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 0fdac5f7-cd43-34c5-5905-dceeb9813952
Version Independent ID: 6fd53f84-dbb2-2a9c-2745-ea1999779113
Content: Embedded analytics to embed Power BI content in your application for your customers - Power BI
Content Source: powerbi-docs/developer/embed-sample-for-customers.md
Service: powerbi
Sub-service: powerbi-developer
GitHub Login: @rkarlin
Microsoft Alias: rkarlin

Hi, @rohancragg -- thanks for your comment! I've assigned it to a few people to look into.

Hi @rohancragg,
We are aware of this limitation.
The plan is to address this limit when Service Principal for Power BI Embedded goes GA and is no longer in "preview."
We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion, and we will reopen the issue.


Carthage Support

I'll definitely do this. Give me a day or two.
Thanks!

Carthage support has been added. Check the README for more details.
Thanks!

thanks


Feature: Migration to Eslint

As tslint is deprecated we need to migrate to Eslint

This is a great point @ruslanguns!
A quick fix to get this in, might be defining the linter here to make sure the app gets generated with that, and here for all the libraries that generated.
If you're up to giving it a shot, more than welcome to review/help merge any PR's.

This is a great point @ruslanguns!
A quick fix to get this in, might be defining the linter here to make sure the app gets generated with that, and here for all the libraries that generated.
If you're up to giving it a shot, more than welcome to review/help merge any PR's.

Btw I am looking at migrating this project to a new Nx 11 structure (where schematics are generators), which would have this by default.

Btw I am looking at migrating this project to a new Nx 11 structure (where schematics are generators), which would have this by default.

This is a great point @ruslanguns!
A quick fix to get this in, might be defining the linter here to make sure the app gets generated with that, and here for all the libraries that generated.
If you're up to giving it a shot, more than welcome, I'm happy to review/help merge any PR's.

Nice feedback, looks like it is just to add a command right there to point to eslint. Will send you the PR soon.

Btw I am looking at migrating this project to a new Nx 11 structure (where schematics are generators), which would have this by default.

I just got noticed about this change, I never use schematics, looks like they are powerful. I will try to learn by using your nxpm stack project there is a beautiful work around.

This is a great point @ruslanguns!
A quick fix to get this in, might be defining the linter here to make sure the app gets generated with that, and here for all the libraries that generated.
If you're up to giving it a shot, more than welcome, I'm happy to review/help merge any PR's.

Nice feedback, looks like it is just to add a command right there to point to eslint. Will send you the PR soon.

Btw I am looking at migrating this project to a new Nx 11 structure (where schematics are generators), which would have this by default.

I just got noticed about this change, I never use schematics, looks like they are powerful. I will try to learn by using your nxpm stack project there is a beautiful work around.


Changes in Builder

Created the simplified update, for situations where we have control of object changes;
Changed in 'where' the origin of the table, which is imported from 'from';

There are no tests, and the implementation duplicated the source code, so will not be accepted. I already change the ORM to do what was tried to be done in this commit.


KeyError: 'model_conf

File "D:\sofa\conda\envs\tgngo\Lib\site-packages\funasr\models\emotion2vec\model.py", line 45, in init
cfg = OmegaConf.create(kwargs["model_conf"])
~~~~~~^^^^^^^^^^^^^^
KeyError: 'model_conf

same. Do u have any solution ?

pip install modelscope to solve

Thanks for fixing this issue, but I think modelscope==1.20.1 is already in the project.toml. Is there any issue in our install guide?

@ltzheng
Please could you help with reducing the memory usage. I am offloading vae to cpu the same way in all other similar projects to reduce memory usage and it works fine.
However this is the first time I am facing problem. The following works and reduce the VRAM usage to 5 GB but inference remains at 0% and never moves. After around 1 hours it times out.
pipeline = VideoPipeline(
        vae=vae,
        reference_net=reference_net,
        diffusion_net=diffusion_net,
        scheduler=noise_scheduler,
        image_proj=image_proj,
    )
    pipeline.to(device=device, dtype=weight_dtype)
    pipeline.vae.to('cpu')

If you can help, I think this will work on low VRAM systems.

@ltzheng Hm no, your env file is ok, I built some custom libraries for my hardware so It caused this problem due to lack of libs


Update eval_keras.py to take a --top_n parameter ...

… for tuning number of results, and to use np.argsort instead of np.argpartition to ensure all results are sorted.

@noodlefrenzy @fujikosu
It's Perfect, @noodlefrenzy!
Let me finish the sorting right away!
Thanks!


Adding kokoro config files for presubmit

Remaining TODOs

[x] Expose test log
[x] Add PHP 5.5 child job
[x] Utilize Xunit xml log
[ ] Make sure the child job links are correct

We currently test against 5.5 as well. Is it possible to add a config for it as well?

This PR is almost ready.
We can utilize the xunit style logs on the UI.
An example php55 test result:
https://source.cloud.google.com/results/invocations/71367833-82ac-4b8b-a5f4-9204e71e4d5f/targets
The last piece is to fix the URLs to the child jobs in the invocation log UI.

The results of the dummy version tests look correct.


Renderer encountered an unexpected error: -2147024809

Windows Terminal version
1.16.2523.0
Windows build number
10.0.19044.1889
Other Software
I'm on an Intel Core i5-2500K CPU, using its integrated HD 3000 GPU. Graphics driver version <IP_ADDRESS>59
Steps to reproduce
Open Windows Terminal
Expected Behavior
No response
Actual Behavior
A "Warning" dialog pops up saying "Renderer encountered an unexpected error: -2147024809".
If I click OK it just pops up again, ad infinitum.
I assume it's related to the Atlas renderer that is now enabled by default in the preview version of Terminal.

@lhecker there's an internal bug for this too. I thought the internal person was on DX but I might have been mistaken

-2147024809 is E_INVALIDARG. Could you run DebugView and check if any failure messages are visible when launching Windows Terminal With AtlasEngine enabled?

@lhecker I might be using it wrong but nothing is logged, except when I close the Terminal app via the task bar ("Suspending").

I think I found the internal bug: MSFT-41000959, but that one is -2005270527 (DXGI_ERROR_INVALID_CALL).
The biggest problem is that the oldest, still functional Intel hardware that I personally have is an i7-6600U, which is clearly far newer than a i5-2500K. I can't really think of any reason why our engine would fail, as I ran it with the reference Direct3D driver all the way down to DirectX 9.1.
@noinkling If you'd like to help with this investigation, another thing you could try is to run dxcpl from the start menu. That application isn't entirely intuitive, but basically you do this:

Afterwards you can modify the DirectX settings for Windows Terminal. For instance:

Just hit "Apply" afterwards and restart Windows Terminal. Once you're done with testing I strongly suggest to do this:

If that doesn't yield any insights, another option would be for you to send us a ETL trace as described here: https://github.com/microsoft/terminal/wiki/Capturing-a-Debug-ETL-Trace
However that requires you to install Windows Performance Recorder (WPR).
It's fine if you don't have the time to investigate this further. But unfortunately it might take a little bit to procure your hardware first.

@lhecker With just the debug setting on, DebugView now shows this message:

[8796] D3D11 ERROR: ID3D11Device::CreatePixelShader: Shader uses features (Raw or Structured Buffers) not supported by this device. Use the CheckFeatureSupport() API with D3D11_FEATURE_D3D10_X_HARDWARE_OPTIONS to discover support for these features if using D3D_FEATURE_LEVEL_10_*. [ STATE_CREATION ERROR #192: CREATEPIXELSHADER_INVALIDSHADERBYTECODE]

Terminal does work when selecting 9_3 or below for the feature level limit. The error happens again when selecting 10_0 or above.
I can confirm that it's supposed to support the DirectX 10.1 feature level, dxdiag and GPU-Z say the same thing.

In the meantime I'd suggest to disable AtlasEngine in the settings tab (there's a switch under "Rendering").

FYI for anyone who can't access the settings due to the error making the Terminal unusable: Edit the settings JSON file manually (see https://docs.microsoft.com/en-us/windows/terminal/install#settings-json-file it's under %LOCALAPPDATA%\Packages\Microsoft.WindowsTerminalPreview_8wekyb3d8bbwe\LocalState\settings.json for the Windows Store release & Terminal Preview) and add:
"defaults": {
  "useAtlasEngine": false
},

into profiles or "useAtlasEngine": false to any profile.

"useAtlasEngine": false

Thank you, this workaround fixed the issue. 😃

What version should this fix appear in? Because I just started encountering this problem today.
Version: 1.16.2530.0

We haven't committed to any specific date yet, but it'll be sometime soon. There's a few more issues around the new text renderer, which we'd like to address as well, but haven't merged yet. This issue will be updated by a bot once the fix has been released.

My windows terminal got infinitely locked up in the Renderer encountered an unexpected error.  So I found this page and applied the suggested fix (in settings.json). Then, however, as I type commands, the terminal app starts to blank out the text entry lines, once they are long enough (so one cannot see the command and its response).
I attach a screenshot where the command sudo service apache2 status has been blanked out by the preview terminal engine.
To me, as a SW developer and engineer, it appears like shaky behavior to go after effects of "shading" and "fading" ... at the expense of breaking the functionality (on lots of combinations of hardware) of a critical app.
My rescue was to go back to the non preview version of the terminal.

To me, as a SW developer and engineer, it appears like shaky behavior to go after effects of "shading" and "fading" ... at the expense of breaking the functionality (on lots of combinations of hardware) of a critical app.
My rescue was to go back to the non preview version of the terminal.

If you did set "useAtlasEngine": false then you re-enabled the same text renderer that has been virtually untouched since at least half a year and is also used in the non-Preview version of Windows Terminal. As such the bug you describe can't be related to the new text renderer and instead sounds like something else entirely. I've created an issue for you: #14027
BTW I'm not sure what you mean with "shading and fading", but if you saw me write "shading" in relation to anything "AtlasEngine", that's just a term to basically describe "colorize text". The new text renderer solves some fundamental performance issues we've had.

I got these two issues after a Windows upgrade had run. I guess it also updated the Terminal Preview app (or the Preview app ran just fine before the Windows update).
For shading / fading - sorry if quickly jumping to conclusions. It is the preview version and there will be issues. I guess I'm fed up with going for "bells and whistles" in various contexts - often then dragging down performance / footprint and reliability.

I guess I'm fed up with going for "bells and whistles" in various contexts - often then dragging down performance / footprint and reliability.

You and me both! Yeah, we've been staying pretty technical when we talk about the "atlas" text engine. At the end of the day, it uses a GPU shader to draw text to the screen as fast as we can produce it. It also happens to support the flashy bells-and-whistles kinds of shaders that you may be thinking of, but even when doing that it's still more performant than the older text rendering engine.
That is, when it works.
I'd love if you could file an issue w/ a video or something showing what you see when the text is flickering. That's definitely something it should never do, and something we'll want to address regardless of which engine is putting text on the screen!

Since no one on the team can reproduce this issue, would anyone be willing to test our upcoming build before we submit it to the Microsoft Store? It's properly signed and if it works it would end up being the build we publish.
Just re-enable AtlasEngine again and open a new tab, or restart the app. But be careful not do to the update while Terminal Preview is already running, as this will potentially force-quit the application during the update.

Windows 10: Microsoft.WindowsTerminalPreview_Win10_1.16.2641.0_8wekyb3d8bbwe.msixbundle
Windows 11: Microsoft.WindowsTerminalPreview_Win11_1.16.2642.0_8wekyb3d8bbwe.msixbundle

@lhecker I can confirm that it fixes the error for me.


In night mode, the text for the code entry dialogs isn't visible

Complete the night mode design by updating the text values for the code entry dialogs so that the text displays in night mode as white.

Fixed in v0.8.3.beta.9


What Windows folders are safe to compact? (LZX)

Tried whole folder - Recovery mode after reboot)

I'm sorry to hear that.  What folder specifically did you try?
Most things other than the base Windows folder should be fairly safe - the Windows folder itself (assuming it's called *:\Windows) and any file specifically tagged with the system flag should be skipped, as mentioned in the README.
Generally I'd recommend limiting it to your Program Files folder, or a Steam folder.  Not things like your user profile.
The Windows folder can be safely compressed using compact.exe via the CompactOS feature - this is specifically not supported by Compactor itself.

It`s OK it was test VM) Ok got it, thanx

compact.exe /CompactOS:always is designed to do this safely.
@Freaky Maybe you wish to expose this via your UI?  N.B.:  this can take hours, so should be done async/in the background.  I've found running it @ realtime priority maxes out disk usage @ 100% (duh), but cpu <4%.


Persisting the connection

On Reload of the page the user always has to reconnect manually.
Is it possible to persist the connection, so that the user if he didn't explicitly remove the dapp from the connected sites
it auto connects?
greets

That is a good point.
This is related to #133.
Added to roadmap.

Easy handle this via the application logic (saving automatic connections preferences in localstorage) and creating a component that automatically calls activate if a connection was previously initialized.
That being said, would be nice too also have this functionality included in the library.

It is scheduled for next milestone.

Is this in a version or something someone can help with?


Tests finding counts at 0

Bug description
All counters at 0 when viewing an engagement's page, under "Tests".

dev on v. 1.5.4 ( 70e29801 )

Hi @madchap . I have found a couple of issues on that engagement page (see https://github.com/DefectDojo/django-DefectDojo/issues/1643 )
Can you check if your issue was already there at https://github.com/DefectDojo/django-DefectDojo/commit/e49bbb5e4fb8cab0d8e2214f153744ed86a8d39a ? If not, your problem might come from  https://github.com/DefectDojo/django-DefectDojo/commit/1720c95bf831236b81c5713d6006b29e55dec33c too.

Yep, I see, I thought of that when I saw them passing by. I'll take a look whenever, this one is not so critical I guess.
Thanks @ptrovatelli !

OK, so tracked it down to view_eng.html at line 196, calling display_tags.py line 677. Effectively, using Active=True, Verified=True. That's just another place to change to adapt to Verified=False, as started.


AndroidUI tests: No views in hierarchy found

Intro
Tests are still shaky they fail once per 10-15 runs
Proposal
Tests mostly fail because of "No views in hierarchy found", screenshots may help to troubleshoot them https://medium.com/stepstone-tech/how-to-capture-screenshots-for-failed-ui-tests-9927eea6e1e4

During debugging tests it seams it may happens when Activity/Fragment is not in RESUMED state so make sense to add checks/waits for this

Reproduces more frequently on API 25

It was produced less frequently during the last builds (usually once in 25-30 builds)
The last one was here https://github.com/hCaptcha/hcaptcha-android-sdk/actions/runs/9776979057/job/27027052421
com.hcaptcha.sdk.HCaptchaDialogFragmentTest > webViewReturnToken[emulator-5554 - 9] FAILED
	androidx.test.espresso.NoMatchingViewException: No views in hierarchy found matching: an instance of android.webkit.WebView and webView.getSettings().getJavaScriptEnabled() is <true>

https://github.com/hCaptcha/hcaptcha-android-sdk/issues/101 looks more annoying at this moment


Can I use “Monitoring Operations” with URQL?

Can I use this with URQL? I see there is an Apollo plugin, but no mention of URQL. Thanks!
https://docs.graphql-hive.com/features/monitoring

ah, makes sense! thanks for clarifying!

@dotansimha I see you don't have a python sdk. Is there an API spec to upload the raw GQL data such that I could build one?

@dotansimha I see you don't have a python sdk. Is there an API spec to upload the raw GQL data such that I could build one?

At the moment we do not have one, but maybe it is worth keeping an eye on this one: https://github.com/kamilkisiela/graphql-hive/issues/1122#issuecomment-1402381597 (and please share your stack and setup, it might help us to prioritize!)

@jerber did you build one for yourself? :)

I don’t believe we did.


Import breaks if Ubuntu 16.04 doesn't have resolvconf directory

Image translation fails if the disk image doesn't have resolvconf directory.
[import-and-translate.translate.translate-disk.wait-for-translator]: 2020-06-13T12:22:12Z WaitForInstancesSignal: Instance "inst-translator-import-and-translate-translate-translate-wdpzn": StatusMatch found: "TranslateStatus: Resetting resolvconf base."
[import-and-translate]: 2020-06-13T12:22:12Z Error running workflow: step "translate" run error: step "translate-disk" run error: step "wait-for-translator" run error: WaitForInstancesSignal FailureMatch found for "inst-translator-import-and-translate-translate-translate-wdpzn": "TranslateFailed: error: sh: /bin/sh: 1: cannot create /etc/resolvconf/resolv.conf.d/base: Directory nonexistent"

Ideally this should check if the directory exists before trying to remove clear the resolvconf configs in daisy_workflows/image_import/ubuntu/translate.py.

A couple of questions:

Did you modify the VM to remove resolveconf or its configs? If not, any idea why they're missing?
Can you post the serial logs from the translate step? You'll find a URL a few lines above the [import-and-translate] log that you posted. It'll look something like this:

CreateInstances: Streaming instance "inst-translator-ubuntu-1604-vmware-translate-test-transl-rxjm8" serial port 1 output to https://storage.cloud.google.com/edens-test-daisy-bkt/daisy-ubuntu-1604-vmware-translate-test-20200619-16:51:57-rxjm8/logs/inst-translator-ubuntu-1604-vmware-translate-test-transl-rxjm8-serial-port1.log

Thanks for taking a look Eric. Reading more about resolvconf, it indeed comes preinstalled since Ubuntu 12.04 but it is possible to uninstall it. In our case, the package had been uninstalled, unfortunately we don't have internal documentation around when and why this was removed for our setup.
In either case, it would be good to check for the existence of the directory/file before trying to clean up the file as is happening in daisy_workflows/image_import/ubuntu/translate.py. To unblock ourselves, we were able to create a dummy directory structure in our VM to avoid this problem.
Happy to send out the small patch to fix this (protecting the code path with g.is_file()) if you think this check should be added.
Please find the serial logs attached.
[1]daisy-import-and-translate-20200613-11_39_44-wdpzn_logs_inst-importer-import-and-translate-import-wdpzn-serial-port1.txt
[2]daisy-import-and-translate-20200613-11_39_44-wdpzn_logs_inst-translator-import-and-translate-translate-translate-wdpzn-serial-port1.txt

Thanks for the info @anuj919 ; I've staged a fix!

Awesome, thanks for the fix!


Exercises-w1

Exercises
Name
Ben Langenberg
Class
202
GitHub username
https://github.com/benl95
Bootcamp exercises
https://github.com/benl95/Opdrachten-week-1
Bootcamp notes

👋 Thanks, we'll look at the exercises from exercises-w2 for your final grade!


0.17 occasionally breaks list rendering

The script then fails with a Type error: n is undefined in this line.
The issue seems to be that sometimes source.observers is a sparse array with a .length greater than the actual length, allowing .pop() to return undefined.
Unfortunately I am not able to provide a simple example where this occurs, but there are apps that break when updating to version 0.17.

Thanks for reporting. I tried to test in every scenario I had but this release saw a rewrite of the reactive core. I will investigate. The observers array should always be push and pop but maybe there is a bug in how items are moved in unsubscribe. Obviously any more details are helpful but I will investigate none the less.

My observation was that including the following check for sparseness after line 347, will only trigger immediately before and when failing:
if (source.observers.length !== source.observers.filter(Boolean).length) {
  console.log(Array.from(source.observers));
}

Thanks I downloaded your app and found an easy to reproduce scenario with the subscribe button. Your last hint is helpful. I see it happening now just trying to figure out how the indexes are getting out of sync.

Ok update. I know what the problem is just deciding on the right fix. Logic was too simple to handle duplicate nodes in the list. In practice checking uniqueness all the time is more expensive than just adding deps that will never run anyway, but obviously if the disposal logic can't handle it we have a problem.

Seems almost too simple of a fix but I realized it is what the old version was doing. Checking node equality was just messing things up.


Shortcut: Fixed selected range and renaming

Fixed an issue where when renaming a shortcut, its extension is cleared.
Fixed selected name range when trying to rename shortcut.
Before:

After:

:shipit:


Documentation:

Documentation:
It may be better if we set the Open API via environment variables.
from skllm.config import SKLLMConfig
from dotenv import load_dotenv, find_dotenv
import os
_ = load_dotenv(find_dotenv())
SKLLMConfig.set_openai_key(os.getenv('OPENAI_API_KEY'))

Hi @Sicolas-Flamel,
I don't think that this is worth including into the documentation. Users should decide for themselves where exactly they want to get the keys from (be it an env variable or anything else).
On the other hand, it is also possible to set the env variable SKLLM_CONFIG_OPENAI_KEY directly (without even using the SKLLMConfig class), and this is something we should probably mention.

OK


add doc for how to track email event data

Description of the change: Add new doc to show how to track event data with Keen IO in the Classroom section
Reason for the change: Help people do cool stuff with event data
Link to original source: https://keen.io/docs/integrations/sendgrid/
@ksigler7

Hello @elof,Thanks again for the PR! We appreciate your contribution and look forward to                                              continued collaboration. Thanks!                                             Team SendGrid DX


checkdb: optimized script and added out-of-bound check

This PR is to solve two issues:

https://github.com/SymbiFlow/prjxray/issues/583: I have noticed that all the tiles are being checked. I have also seen that all the tiles with the same tile_type share the same bits, making it highly time consuming if there is the need to check every single one of them. This way the script is much faster, but I am not sure it is the best solution, because maybe some tiles of the same type have differences.
https://github.com/SymbiFlow/prjxray/issues/575: during the extraction of the bits there is an assertion that checks whether the bit is consistent with the frames number of the tile.

Signed-off-by: Alessandro Comodi<EMAIL_ADDRESS>

I discovered that the comparison between the already checked tiles and the new ones was the bottleneck of the script. By modifying the comparison mechanism it should take much less now


Backend for Submit announcement form

I gave a though to it and I am implementing it using nodemailer. Just opened an issue so that you may add this to you board.

#39 Closes this


Tooltips overlap

http://imgur.com/S1bQhhN
I have a lot of datapoints and my tooltips overlap.

I have far fewer tooltips but I'd like them permanently displayed which works well except when they're perfectly in line in which case the problem exists for me too (albeit to a lesser degree).

I've got the same issue. I did notice, however, that this is only an issue if there's a single dataset.
Here's a jsbin. If you remove the commented second dataset, you'd see only one multi-tooltip. I think if it's solved there, it should be done the same way with the single tooltip.

Hello, today I am closing all issues that are only affecting version 1 of Chart.js as WONTFIX.
If this issue does affect version 2 as well, I apologize for the error. Please create a test case against Chart.js 2 using on of the below websites and we will be happy to reopen the issue and update its classification:

http://jsbin.com/
http://jsfiddle.net/
http://codepen.io/pen/
Premade template: http://codepen.io/pen?template=JXVYzq


[Airflow KubernetesExecutor] Not working Entrypoint on worker pod

Hi, guys.
I'm trying to migrating airflow 1.10.4 for KubernetesExecutor.
I use same image for webserver, schedular, worker.
But, as some reason, with creating KubernetesExecutor pod, entrypoint.sh doesn't execute.
Of Course, webserver, schedular works fine.
I dig this problem for over 1 week, I can't find the reason why they don't execute entrypoint.sh.
Dockerfile
...
ARG AIRFLOW_VERSION=1.10.4
ARG AIRFLOW_USER_HOME=/usr/local/airflow
...
COPY ./entrypoint.sh /entrypoint.sh
COPY ./dags ${AIRFLOW_HOME}/dags

RUN chown -R airflow: ${AIRFLOW_USER_HOME}
RUN chmod +x /entrypoint.sh

USER airflow
WORKDIR ${AIRFLOW_USER_HOME}
ENTRYPOINT ["/entrypoint.sh"]
CMD ["--help"]

entrypoint.sh
#!/usr/bin/env bash

ECHO "START"
TRY_LOOP="10"
AIRFLOW_HOME="/usr/local/airflow"
...
# Defaults and back-compat
: "${AIRFLOW__CORE__FERNET_KEY:=${FERNET_KEY:=$(python -c "from cryptography.fernet import Fernet; FERNET_KEY = Fernet.generate_key().decode(); print(FERNET_KEY)")}}"
...
case "$1" in
  webserver)
    airflow initdb
    if [ "$AIRFLOW__CORE__EXECUTOR" = "LocalExecutor" ]; then
      # With the "Local" executor it should all run in one container.
      airflow scheduler &
    fi
    exec airflow webserver
    ;;
  worker|scheduler)
    # To give the webserver time to run initdb.
    sleep 10
    exec airflow "$@"
    ;;
  flower)
    sleep 10
    exec airflow "$@"
    ;;
  version)
    exec airflow "$@"
    ;;
  *)
    # The command is something like bash, not an airflow subcommand. Just run it in the right environment.
    exec "$@"
    ;;
esac

And FYI, I use airflow.cfg as k8s configmap

I also have this same question. I was wondering if this line:
if [ "$AIRFLOW__CORE__EXECUTOR" = "LocalExecutor" ]
be:
if [ "$AIRFLOW__CORE__EXECUTOR" = "KubernetesExecutor" ]
with the kuEx you should only have 1 webserver pod that spins up tasks per dag as needed in other containers.

You need to run 'pip install apache-airflow[kubernetes]==${AIRFLOW_VERSION}` in your dockerfile, it's not installed in this container by default.

I have already pip installed the correct version of airflow with all the req'd airflow packages. I am successfully able to spin up  pods with the Kubernetes Operator, however now my jobs fail because there is no log file. I am gettign this Error in airflow: *** Log file does not exist: /usr/local/airflow/logs/hw_load.load_and_validate_dimensions/exrt_load_dimension_HWLOAD_UNIT/2019-10-16T16:17:37.011639+00:00/1.log *** Fetching from: http://hwloadsystemloadandvalidatedimensions-790ab2287dc5454f97d70d03b:8793/log/hw_load.load_and_validate_dimensions/exrt_load_dimension_HWLOAD/2019-10-16T16:17:37.011639+00:00/1.log *** Failed to fetch log file from worker. HTTPConnectionPool(host='hwloadsystemloadandvalidatedimensions-790ab2287dc5454f97d70d03b', port=8793): Max retries exceeded with url: /log/hw_load.load_and_validate_dimensions/exrt_load_dimension_HWLOAD/2019-10-16T16:17:37.011639+00:00/1.log (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0615ce4f28>: Failed to establish a new connection: [Errno -2] Name or service not known',))

@ketikat can you share how you got this to work? I have installed the dependencies and have tried to alter the entrypoint.sh script but it nonetheless keeps starting with the SequentialExecutor.

here is my airflow entrypoint.sh for reference:
`#!/usr/bin/env bash
TRY_LOOP="20"
: "${AIRFLOW_HOME:="/usr/local/airflow"}"
: "${REDIS_HOST:="redis"}"
: "${REDIS_PORT:="6379"}"
: "${REDIS_PASSWORD:=""}"
: "${DB_HOST:="mysql-server"}"
: "${DB_PORT:="3306"}"
export
AIRFLOW__CORE__EXECUTOR="KubernetesExecutor"
AIRFLOW__CORE__FERNET_KEY=$FERNET_KEY
AIRFLOW_HOME="/usr/local/airflow"
AIRFLOW__CORE__DAGS_FOLDER="/usr/local/airflow/mydags"
DAGS_FOLDER="/usr/local/airflow/mydags"
AIRFLOW__WEBSERVER__RBAC="True"
wait_for_port() {
local name="$1" host="$2" port="$3"
local j=0
while ! nc -z "$host" "$port" >/dev/null 2>&1 < /dev/null; do
j=$((j+1))
if [ $j -ge $TRY_LOOP ]; then
echo >&2 "---- waiting for port: $(date) - $host:$port still not reachable, giving up ---- "
exit 1
fi
echo "---- waiting for port: $(date) - waiting for $name... $j/$TRY_LOOP ---- "
sleep 5
done
}
wait_for_redis() {
if [ "$AIRFLOW__CORE__EXECUTOR" = "CeleryExecutor" ]
echo "---- Entrypoint: Waiting  for Redis to start ---- "
then
wait_for_port "Redis" "$REDIS_HOST" "$REDIS_PORT"
fi
}
wait_for_mysql() {
echo "---- Entrypoint: Waiting  for mysql server to start ---- "
wait_for_port "mysql-server" "$DB_HOST" "$DB_PORT"
}
if [ "$AIRFLOW__CORE__EXECUTOR" = "CeleryExecutor" ]; then
export AIRFLOW__CORE__SQL_ALCHEMY_CONN="mysql://$AIRFLOW_DB_USER:$AIRFLOW_DB_PASSWORD@mysql-server:3306/airflow"
export AIRFLOW__CELERY__RESULT_BACKEND="db+mysql://$AIRFLOW_DB_USER:$AIRFLOW_DB_PASSWORD@mysql-server:3306/airflow"
wait_for_mysql
export AIRFLOW__CELERY__BROKER_URL="redis://redis:6379/1"
wait_for_redis
fi
if [ "$AIRFLOW__CORE__EXECUTOR" = "KubernetesExecutor" ]; then
export AIRFLOW__CORE__SQL_ALCHEMY_CONN="mysql://$AIRFLOW_DB_USER:$AIRFLOW_DB_PASSWORD@mysql-server:3306/airflow"
wait_for_mysql
fi
case "$1" in
webserver)
echo "---- Entrypoint: IN WEBSERVER ----"
wait_for_mysql
airflow upgradedb
echo "---- Entrypoint: airflow db is intialized ---- "
echo "---- Entrypoint: Creating Admin account for Admin User ---- "
# this will be the new syntax in next airflow upgrade:
# airflow users --create --username $AIRFLOW_USERNAME  --email $AIRFLOW_EMAIL --password $AIRFLOW_PASSWORD #--role Admin
# this is the current way of doing it:
airflow create_user -r Admin -u $AIRFLOW_USERNAME -e $AIRFLOW_EMAIL -p $AIRFLOW_PASSWORD -f Admin -l User

if [ "$AIRFLOW__CORE__EXECUTOR" = "KubernetesExecutor" ]; then
  # With the "Kubernetes" executor it should all run in one container.
  airflow scheduler &
fi

if [ "$AIRFLOW__CORE__EXECUTOR" = "LocalExecutor" ]; then
  # With the "Local" executor it should all run in one container.
  airflow scheduler &
fi

sleep 10
exec airflow webserver
;;

scheduler)
echo "---- Entrypoint: IN SCHEDULER ----"
# https://github.ibm.com/map-ds/MAP_DevOps/blob/59af8f3bb0b7bc06d7eddc5fc4dcfed829437dfe/etl-docker/entrypoint.sh
wait_for_mysql
wait_for_redis
sleep 10
exec airflow "$@"
;;
worker)
echo "---- Entrypoint: IN WORKER ----"
wait_for_mysql
wait_for_redis
sleep 10
exec airflow worker
exec airflow "$@"
;;
flower)
echo "---- Entrypoint: IN FLOWER ----"
sleep 10
exec airflow "$@"
;;
*)
exec "$@"
;;
esac`

@tblazina

Thanks @ketikat!

I have the same issue, running the entrypoint script on worker pods, any fixes for this?


kubectl: Handle splitting host:port when no port is present

In the proxy filter, if listening on port 80 or 443 there will be no port present in the Host: HTTP header. Just use the entire thing in these cases, and don't reject requests when no port is present in the Host: HTTP header.

Example:
$ su -m root -c 'kubectl proxy --v=3 --port=80 --api-prefix=/'
Password:
Starting to serve on localhost:80
I0702 14:41:14.035276   16224 proxy_server.go:100] Filter rejecting GET /api/v1/services

The request fails:
$ curl http://localhost/api/v1/services

Unauthorized

After the patch:
$ curl http://localhost/api/v1/services
{
  "kind": "ServiceList",
  "apiVersion": "v1",
  "metadata": {
    "selfLink": "/api/v1/services",
    "resourceVersion": "151"
  },
  "items": [
    {
      "metadata": {
...

@eparis When you have time could you review this bug fix?

cc @bgrant0607 @jlowdermilk as potential reviewers

ok to test

LGTM

LGTM

Daniel as second lead for lgtm

Added unit test.

Added comment, fixed up naming, made private symbol.

@k8s-bot test this please

add to whitelist


Disabling Log Shipping Functionality

Summary
today at 06:49:42 /var/run/s6/etc/cont-init.d/05-logging: line 11: /assets/cron//logrotate: No such file or directory
today at 06:49:42 2021-08-09-06:49:42 [NOTICE] ** [logging] Disabling Log Shipping Functionality
Steps to reproduce
Start container, see logs.
What is the expected correct behavior?
For log rotate to work.
Relevant logs and/or screenshots
today at 06:49:41 2021-08-09-06:49:41 [NOTICE] ** [container-init] Detected Container that has been restarted - Cleaning '/tmp/state' files
today at 06:49:41 [cont-init.d] 00-startup: exited 0.
today at 06:49:41 [cont-init.d] 01-timezone: executing...
today at 06:49:41 [cont-init.d] 01-timezone: exited 0.
today at 06:49:41 [cont-init.d] 02-permissions: executing...
today at 06:49:41 [cont-init.d] 02-permissions: exited 0.
today at 06:49:41 [cont-init.d] 03-monitoring: executing...
today at 06:49:41 2021-08-09-06:49:41 [NOTICE] ** [monitoring] Container configured for monitoring with 'zabbix modern'
today at 06:49:41 [cont-init.d] 03-monitoring: exited 0.
today at 06:49:41 [cont-init.d] 04-scheduling: executing...
today at 06:49:41 2021-08-09-06:49:41 [NOTICE] ** [scheduling] Disabling Scheduling
today at 06:49:41 [cont-init.d] 04-scheduling: exited 0.
today at 06:49:41 [cont-init.d] 05-logging: executing...
today at 06:49:42 /var/run/s6/etc/cont-init.d/05-logging: line 11: /assets/cron//logrotate: No such file or directory
today at 06:49:42 2021-08-09-06:49:42 [NOTICE] ** [logging] Disabling Log Shipping Functionality
today at 06:49:42 [cont-init.d] 05-logging: exited 0.
today at 06:49:42 [cont-init.d] 06-messaging: executing...
today at 06:49:42 2021-08-09-06:49:42 [NOTICE] ** [messaging] Container configured to route mail via SMTP to 'postfix-relay'
today at 06:49:42 [cont-init.d] 06-messaging: exited 0.
today at 06:49:42 [cont-init.d] 10-cloudflare-companion: executing...
today at 06:49:42 [cont-init.d] 10-cloudflare-companion: exited 0.
today at 06:49:42 [cont-init.d] 99-container: executing...
today at 06:49:42 [cont-init.d] 99-container: exited 0.
today at 06:49:42 [cont-init.d] done.
today at 06:49:42 [services.d] starting services
today at 06:49:42 [services.d] done.

Environment
Docker, latest
  # Cloudflare-Companion - Automatic CNAME DNS Creation
  cfcompanion:
    container_name: CF-Companion
    image: tiredofit/traefik-cloudflare-companion:latest
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      - TIMEZONE=$TZ
      - TRAEFIK_VERSION=2
      - CF_EMAIL=$CLOUDFLARE_EMAIL # Same as traefik
      # - CF_TOKEN=$CLOUDFLARE_API_TOKEN # Scoped api token not working. Error 10000.
      - CF_TOKEN=$CLOUDFLARE_API_KEY # Same as traefik
      - TARGET_DOMAIN=$DOMAINNAME
      - DOMAIN1=$DOMAINNAME
      - DOMAIN1_ZONE_ID=$CLOUDFLARE_ZONEID # Copy from Cloudflare Overview page
      - DOMAIN1_PROXIED=TRUE
    depends_on:
      - dnscrypt-proxy
    # depends_on:
    #   dnscrypt-proxy:
    #     condition: service_healthy
    labels:
      # Add hosts specified in rules here to force cf-companion to create the CNAMEs
      # Since cf-companion creates CNAMEs based on host rules, this a workaround for non-docker/external apps
      - "traefik.http.routers.cf-companion-rtr.rule=Host(`netdata.$DOMAINNAME`)"
      #- "traefik.http.routers.cf-companion-rtr.rule=Host(`impi.$DOMAINNAME`)" #TLS middlewares not yet implemented https://github.com/containous/traefik/issues/6684
      #- "traefik.http.routers.cf-companion-rtr.rule=Host(`webmin.$DOMAINNAME`)" #TLS middlewares not yet implemented https://github.com/containous/traefik/issues/6684
    networks:
      - isonet

Possible fixes
Looks like line 11 has an errant "/" (/assets/cron//logrotate)

Can you try CONTAINER_ENABLE_SHCEDULING=TRUE as an environment variable? That should turn back on Cron but somewhere my logic for log shipping/log rotation has getting broken in the base image. I will investigate.

In the meantime I've also pushed 6.6.2 which should negate having to add that environment variable

The extra "/' is a ~dirty~ hack just to catch if someone has changed the "base" prefix of where cron files are pulled from without putting a trailing "/" at the end of their path . Theoretically it shouldn't affect anything, just looks a bit wonky.

Heya,
Both with new image + container_enable_scheduling:
today at 19:41:45 [s6-init] making user provided files available at /var/run/s6/etc...exited 0.
today at 19:41:45 [s6-init] ensuring user provided files have correct perms...exited 0.
today at 19:41:45 [fix-attrs.d] applying ownership & permissions fixes...
today at 19:41:45 [fix-attrs.d] 00-functions: applying...
today at 19:41:45 [fix-attrs.d] 00-functions: exited 0.
today at 19:41:45 [fix-attrs.d] 01-s6: applying...
today at 19:41:46 [fix-attrs.d] 01-s6: exited 0.
today at 19:41:46 [fix-attrs.d] 02-zabbix: applying...
today at 19:41:46 [fix-attrs.d] 02-zabbix: exited 0.
today at 19:41:46 [fix-attrs.d] 03-logrotate: applying...
today at 19:41:46 [fix-attrs.d] 03-logrotate: exited 0.
today at 19:41:46 [fix-attrs.d] done.
today at 19:41:46 [cont-init.d] executing container initialization scripts...
today at 19:41:46 [cont-init.d] 00-startup: executing...
today at 19:41:46 [cont-init.d] 00-startup: exited 0.
today at 19:41:46 [cont-init.d] 01-timezone: executing...
today at 19:41:46 2021-08-09-18:41:46 [NOTICE] ** [timezone] Timezone: Setting to 'Europe/London' from 'Etc/GMT'
today at 19:41:46 [cont-init.d] 01-timezone: exited 0.
today at 19:41:46 [cont-init.d] 02-permissions: executing...
today at 19:41:46 [cont-init.d] 02-permissions: exited 0.
today at 19:41:46 [cont-init.d] 03-monitoring: executing...
today at 19:41:46 2021-08-09-19:41:46 [NOTICE] ** [monitoring] Container configured for monitoring with 'zabbix modern'
today at 19:41:46 [cont-init.d] 03-monitoring: exited 0.
today at 19:41:46 [cont-init.d] 04-scheduling: executing...
today at 19:41:46 2021-08-09-19:41:46 [NOTICE] ** [scheduling] Disabling Scheduling
today at 19:41:46 [cont-init.d] 04-scheduling: exited 0.
today at 19:41:46 [cont-init.d] 05-logging: executing...
today at 19:41:46 /var/run/s6/etc/cont-init.d/05-logging: line 11: /assets/cron//logrotate: No such file or directory
today at 19:41:46 2021-08-09-19:41:46 [NOTICE] ** [logging] Disabling Log Shipping Functionality
today at 19:41:46 [cont-init.d] 05-logging: exited 0.
today at 19:41:46 [cont-init.d] 06-messaging: executing...
today at 19:41:46 2021-08-09-19:41:46 [NOTICE] ** [messaging] Container configured to route mail via SMTP to 'postfix-relay'
today at 19:41:46 [cont-init.d] 06-messaging: exited 0.
today at 19:41:46 [cont-init.d] 10-cloudflare-companion: executing...
today at 19:41:46 [cont-init.d] 10-cloudflare-companion: exited 0.
today at 19:41:46 [cont-init.d] 99-container: executing...
today at 19:41:46 [cont-init.d] 99-container: exited 0.
today at 19:41:46 [cont-init.d] done.
today at 19:41:46 [services.d] starting services
today at 19:41:46 [services.d] done.

Just something I picked up whilst looking at the logs! Core functionality is still working as expected which is good!
  # Cloudflare-Companion - Automatic CNAME DNS Creation
  cfcompanion:
    container_name: CF-Companion
    image: tiredofit/traefik-cloudflare-companion:latest
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      - TIMEZONE=$TZ
      - TRAEFIK_VERSION=2
      - CF_EMAIL=$CLOUDFLARE_EMAIL # Same as traefik
      # - CF_TOKEN=$CLOUDFLARE_API_TOKEN # Scoped api token not working. Error 10000.
      - CF_TOKEN=$CLOUDFLARE_API_KEY # Same as traefik
      - TARGET_DOMAIN=$DOMAINNAME
      - DOMAIN1=$DOMAINNAME
      - DOMAIN1_ZONE_ID=$CLOUDFLARE_ZONEID # Copy from Cloudflare Overview page
      - DOMAIN1_PROXIED=TRUE
      - CONTAINER_ENABLE_SHCEDULING=TRUE
    depends_on:
      - dnscrypt-proxy
    labels:
      # Add hosts specified in rules here to force cf-companion to create the CNAMEs
      # Since cf-companion creates CNAMEs based on host rules, this a workaround for non-docker/external apps
      - "traefik.http.routers.cf-companion-rtr.rule=Host(`netdata.$DOMAINNAME`)"
      #- "traefik.http.routers.cf-companion-rtr.rule=Host(`impi.$DOMAINNAME`)" #TLS middlewares not yet implemented https://github.com/containous/traefik/issues/6684
      #- "traefik.http.routers.cf-companion-rtr.rule=Host(`webmin.$DOMAINNAME`)" #TLS middlewares not yet implemented https://github.com/containous/traefik/issues/6684
    networks:
      - isonet

OK! I'm taking a day off today but I will look into this. I had a client request big changes to my base images and it seems I have a few kinks to work out. Overall functionality of the image should not be affected are you have noticed, I just need to untangle some of the technical debt I've added.

No worries man! Zero rush on this, as I said, base functionality works as intended, this was just something that was picked up by chance, and realistically shouldn't make any massive impact.
More an FYI rather than anything else! Have a good day off!


document GLUT install process

resolves #31

Thanks! Just a heads-up: once we switch over to the new installers, the install directories will be different, namely c:\Users\username\AppData\Local\minghc-7.10.1-i386, for example. Would you be able to send a new PR to include those paths?

Yea no problem.


Update image card hotspot styling

Summary

updated styling for hotspots

:tada: This PR is included in version 1.90.2 :tada:
The release is available on:

npm package (@release-1.x dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 2.5.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


fix: fix telemetry queries

Changes

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#1277  👈
#1276
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @NathanFlurry and the rest of your teammates on  Graphite


Unexpected number of iterations with multiprocess minibatch loading

🐛 Bugs / Unexpected behaviors

If a dataset prepared as a LabeledVideoDataset instance is loaded with num_workers > 1 and batch_size > 1, the number of batches loaded is different from what is expected, which is num_videos // batch_size with a single-view, single-clip-per-video dataset and drop_last=True. With drop_last=False, DataLoader yields unexpectedly many batches with the sizes less than batch_size. Apparently, the number of iterations is fixed to a multiple of num_workers.
This is generally not very crutial; it just results in the slightly smaller number of training iterations per epoch and the slightly more eval iterations. But it can cause some trouble when the number of iterations needs to be guessed.
Instructions To Reproduce the Issue:
Code to reproduce the issue:
import os

from pytorchvideo.data import LabeledVideoDataset
from pytorchvideo.data.clip_sampling import RandomClipSampler
from pytorchvideo.transforms import ApplyTransformToKey, UniformTemporalSubsample
from torch.utils.data import DataLoader
from torchvision.transforms import Compose, RandomCrop

num_videos = 24

# generate labeled_video_path with num_videos==24.
data_dir = "/path/to/dir/with/many/videos"
files = os.listdir(data_dir)
labeled_video_path = [
    (os.path.join(data_dir, files[i]), {"label": 0}) for i in range(num_videos)
]

transform = ApplyTransformToKey(  # just to make sure that auto collation works properly
    key="video", transform=Compose([UniformTemporalSubsample(8), RandomCrop(32)])
)
dataset = LabeledVideoDataset(
    labeled_video_path,
    clip_sampler=RandomClipSampler(2),  # single clip per video
    transform=transform,
    decode_audio=False,
)

for num_workers in (8, 1):
    for batch_size in (2, 1):
        for drop_last in (True, False):
            loader = DataLoader(
                dataset,
                batch_size=batch_size,
                num_workers=num_workers,
                drop_last=drop_last,
            )
            num_iters = len([batch for batch in loader])
            print(
                f"ACTUAL: {num_iters}, EXPECTED: {num_videos // batch_size} "
                f"(batch_size={batch_size}, num_workers={num_workers}, drop_last={drop_last})"
            )

Output:
ACTUAL: 8, EXPECTED: 12 (batch_size=2, num_workers=8, drop_last=True)
ACTUAL: 16, EXPECTED: 12 (batch_size=2, num_workers=8, drop_last=False)
ACTUAL: 24, EXPECTED: 24 (batch_size=1, num_workers=8, drop_last=True)
ACTUAL: 24, EXPECTED: 24 (batch_size=1, num_workers=8, drop_last=False)
ACTUAL: 12, EXPECTED: 12 (batch_size=2, num_workers=1, drop_last=True)
ACTUAL: 12, EXPECTED: 12 (batch_size=2, num_workers=1, drop_last=False)
ACTUAL: 24, EXPECTED: 24 (batch_size=1, num_workers=1, drop_last=True)
ACTUAL: 24, EXPECTED: 24 (batch_size=1, num_workers=1, drop_last=False)

I think it is expected as the LabeledVideoDataset inherits IterableDataset from Pytorch.
In Pytorch documentation on data I found this note:

When fetching from iterable-style datasets with multi-processing, the drop_last argument drops the last non-full batch of each worker’s dataset replica.

I think that in order to have the right amount of batches, the labeled video datasets should follow map-style datasets. I understand why the iterable-style dataset is interesting especially when the stream of data is untrustworthy but it also has its downsides.

@juliendenize No, it's not a problem inherent in IterableDataset, but in how LabeledVideoDataset is implemented. Note the following warning on the same page:

len(dataloader) heuristic is based on the length of the sampler used. When dataset is an IterableDataset, it instead returns an estimate based on len(dataset) / batch_size, with proper rounding depending on drop_last, regardless of multi-process loading configurations. This represents the best guess PyTorch can make because PyTorch trusts user dataset code in correctly handling multi-process loading to avoid duplicate data.
However, if sharding results in multiple workers having incomplete last batches, this estimate can still be inaccurate, because (1) an otherwise complete batch can be broken into multiple ones and (2) more than one batch worth of samples can be dropped when drop_last is set. Unfortunately, PyTorch can not detect such cases in general.

I think the second paragraph describes exactly what is happening now. Current implementation of LabeledVideoDataset distributes data evenly to the replicas, resulting in multiple workers having incomplete last batches. One can alternatively implement LabeledVideoDataset so that it assign to all but at most one replicas data whose length is a multiple of the batch size. Then it should behave as expected. In this case, some workers might produce one more batch than the others, but that's OK.
I would say that len(dataloader) is always expected to be len(dataset) / batch_size with proper rounding. It is just an expectation and not a requirement, but fulfilling it is desirable when it's possible.

Yes, you are right, I'm currently trying to implement a LabeledVideoDataset in my project to alleviate this issue, but I didn't think of a proper way to pass the batch size to the MultiProcessSampler as it is called by the dataset and I don't like the idea to pass the batch_size to the dataset.
I think I will follow PySlowFast repo Kinetics implementation and implement a map-style dataset. As far as I understand it, it doesn't require a lot of changes in LabeledVideoDataset and it's nice to have __getitem__ and __len__ implemented. It should also remove the need to specify how the workers operate. Also, it would be easier to use pytorch-lightning without requiring to specify the current epoch for fixing seed at each epoch for example. I don't know the downsides of using this kind of datasets in comparison with Iterable though.


Remove Voice channel from JDA cache on deletion

Pull Request Etiquette

[x] I have checked the PRs for upcoming features/bug fixes.
[x] I have read the contributing guidelines.

Changes

[x] Internal code
[ ] Library interface (affecting end-user code)
[ ] Documentation
[ ] Other: _____

Description
When a voice channel is deleted on discord the handler only ever deleted it from the guild cache and not from the JDA cache. This caused an exception being thrown when getting the guild from a voice channel via the JDA cache because the guild was garbage collected.
This pull request fixes that by also deleting the voice channel from the JDA cache.

I fixed this a year ago but it seems to have been re-introduced by v4 somewhere along the way.

This is strange as i forked lazy-loading branch before v4 has been released and this bug seems to be fixed in the lazy loading branch instead i guess?


OLS-89: Unit tests: missing credentials in config

Description
Unit tests: missing credentials in config
Type of change

[ ] Refactor
[ ] New feature
[ ] Bug fix
[ ] CVE fix
[ ] Optimization
[ ] Documentation Update
[ ] Configuration Update
[ ] Bump-up dependent library
[x] Unit tests

Related Tickets & Documents

Related Issue #OLS-89

Checklist before requesting a review

[x] I have performed a self-review of my code.
[x] PR has passed all pre-merge test jobs.
[ ] If it is a core feature, I have added thorough tests.

/lgtm

/approve


Caused by: java.lang.UnsatisfiedLinkError: no zstd-jni in java.library.path

Hi,
We are getting the following error messages and we verified that the libzstd-jni.so file exists. Could you help us to resolve this issue.
Caused by: org.apache.kafka.common.KafkaException: java.lang.UnsatisfiedLinkError: no zstd-jni in java.library.path
Unsupported OS/arch, cannot find /linux/amd64/libzstd-jni.so or load zstd-jni from system libraries. Please try building from source the jar or providing libzstd-jni in your system.
at org.apache.kafka.common.record.CompressionType$5.wrapForInput(CompressionType.java:133)
at org.apache.kafka.common.record.DefaultRecordBatch.compressedIterator(DefaultRecordBatch.java:261)
at org.apache.kafka.common.record.DefaultRecordBatch.streamingIterator(DefaultRecordBatch.java:346)
at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.nextFetchedRecord(Fetcher.java:1496)
at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.fetchRecords(Fetcher.java:1533)
at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.access$1700(Fetcher.java:1374)
at org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords(Fetcher.java:676)
at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:631)
at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1313)
... 8 more
Caused by: java.lang.UnsatisfiedLinkError: no zstd-jni in java.library.path
Unsupported OS/arch, cannot find /linux/amd64/libzstd-jni.so or load zstd-jni from system libraries. Please try building from source the jar or providing libzstd-jni in your system.
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1865)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at com.github.luben.zstd.util.Native.load(Native.java:108)
at com.github.luben.zstd.util.Native.load(Native.java:60)
at com.github.luben.zstd.ZstdInputStream.(ZstdInputStream.java:21)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.kafka.common.record.CompressionType.findConstructor(CompressionType.java:226)
at org.apache.kafka.common.record.CompressionType.access$100(CompressionType.java:37)
at org.apache.kafka.common.record.CompressionType$ZstdConstructors.(CompressionType.java:218)
at org.apache.kafka.common.record.CompressionType$5.wrapForInput(CompressionType.java:131)
... 16 more

Any chance you have a mix of old/new versions of zstd-jni on the classpath?

I think this is related to an older issue I was reporting. Unfortunately at that moment I was not able to try other alternatives, so we postponed the implementation of the feature, but now we are facing this issue again.
#https://github.com/luben/zstd-jni/issues/162

Can you try with the newer versions of zstd-jni?

Just tried with https://mvnrepository.com/artifact/com.github.luben/zstd-jni/1.5.0-4 and still facing the issue.

It still looks like some mixing of versions. Previously the native library didn't include the version in the file name, the library was named  /linux/amd64/libzstd-jni.so. Since https://github.com/luben/zstd-jni/pull/169 the native library is also versioned. It looks like the java code that loads the native library is older (no versioning) but the jar supplies only the versioned lib.

I have solved this issue by implementing these 2 apis: http://ip-api.com/json and https://timezonedb.com/

@rengaa30, any chance you use some kind of jar minimization/repackaging/relocation? If you do this, don't rename the zstd classes and also make sure to include also the native libraries. Better just exclude the zstd-jni from the repackaging.

@luben
getting this error while trying to write the data as "delta" format with pyspark . we have older version in /usr/lib64/libzstd-jni-1.4.4.so file, but we are trying to use libzstd-jni-1.5.5-4.jar ,then we are getting the following error.
if we downgrade the version then we are getting some of classes not found like "" java.lang.NoClassDefFoundError: com/github/luben/zstd/ZstdOutputStreamNoFinalizer"
Kindly advice , how can I fix this issue
ERROR:
24/10/02 00:39:50 ERROR MapOutputTrackerMaster: null
java.lang.reflect.InvocationTargetException
Caused by: java.lang.UnsatisfiedLinkError: /tmp/libzstd-jni-1.5.5-42506242103565835660.so: /tmp/libzstd-jni-1.5.5-42506242103565835660.so: failed to map segment from shared object
no zstd-jni-1.5.5-4 in java.library.path
Unsupported OS/arch, cannot find /linux/amd64/libzstd-jni-1.5.5-4.so or load zstd-jni-1.5.5-4 from system libraries. Please try building from source the jar or providing libzstd-jni-1.5.5-4 in your system.
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860)

@luben
I am also facing the same error while running a pyspark job even though zstd-jni-1.5.6-6.jar exists in the path, LIBRARY_PATH.
24/10/27 04:32:04 ERROR MapOutputTrackerMaster: null
java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
	at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
	at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:87)
	at org.apache.spark.MapOutputTracker$.serializeOutputStatuses(MapOutputTracker.scala:1492)
	at org.apache.spark.ShuffleStatus.$anonfun$serializedMapStatus$2(MapOutputTracker.scala:338)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.ShuffleStatus.withWriteLock(MapOutputTracker.scala:77)
	at org.apache.spark.ShuffleStatus.serializedMapStatus(MapOutputTracker.scala:335)
	at org.apache.spark.MapOutputTrackerMaster$MessageLoop.handleStatusMessage(MapOutputTracker.scala:729)
	at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:746)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.UnsatisfiedLinkError: /tmp/libzstd-jni-1.5.6-61001848644097109181.so: /tmp/libzstd-jni-1.5.6-61001848644097109181.so: failed to map segment from shared object
no zstd-jni-1.5.6-6 in java.library.path: [/opt/spark/jars]
Unsupported OS/arch, cannot find /linux/amd64/libzstd-jni-1.5.6-6.so or load zstd-jni-1.5.6-6 from system libraries. Please try building from source the jar or providing libzstd-jni-1.5.6-6 in your system.
	at java.base/java.lang.ClassLoader.loadLibrary(Unknown Source)
	at java.base/java.lang.Runtime.loadLibrary0(Unknown Source)

@naveenbussari , I think this is another issue. Check you check that /tmp is not mounted with noexec? If that's the case you can specify what directory to use for extracting the native lib by defining property ZstdTempFolder

Thanks @luben
/tmp is mounted with noexec. Fixed it by mounting to some other location.


KAFKA-9171: Handle ReplicaNotAvailableException during DelayedFetch

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@ijuma Thanks for the review. Have confirmed that none of the exceptions in the catch block are subclasses of another exception in the catch block.

@ijuma Thanks for the review, merging to trunk and 2.4.


[backoffice] product disappear from list products after performing edit this product

Can you add detail description ?
best idea to add step by step to reproduce

At backoffice, after clicking edit a product, you'll be navigated to product lists and can't see this product anymore.
But you still can see this product with changed information in storefront.
Right now I don't have any clue about this issue. Anybody hands on this issue should make an effort to figure out why it happens.


getDataEntriesForSession() is slow when retrieving 200+ nodes

Yes... but make it faster!

Updated to use a server-side call.

https://github.com/GearsAD/GraffSDK.jl/blob/master/src/services/SessionService.jl#L290-L322


"run scam" button

instead of the small cog button in the bottom left corner, there should be a big red "run SCAM" button in red at the top or bottom

done: https://scamqc.bdrc.io/?folder=LTWA2023withacc%2F000950%2F

perfect, thanks!


Improved error message in FixedChannelPool

Motivation:
FixedChannelPool allows users to configure acquireTimeoutMillis and expects given value to be greater or equal to zero when timeout action is supplied. However, validation error message said that value is expected to be greater or equal to one. Code performs check against zero.
Modifications:
Changed error message to say that value greater or equal to zero is expected. Added test to check that zero is an acceptable value.
Result:
Exception with right error message is thrown.

Cherry-picked into 4.1 (25d146038cd825311b93bbabeb3f979796bd71d3) and 4.0 (33d7348fdab117ef7b20f8a150db33acd8f6c223).
@lutovich thanks!


Support Glance

mapping file plus engine enhancements if needed

Do you supply glance audit yaml? I test nova, neutron and cinder, it works well.

Glance and Swift is missing yet, looking for contributors :-)

@like-inspur Glance is now added.


Specification for prefix versions broke

Issue type
Wrong or misleading information
Problem description
The page mentions the + character, but it's not listed at all in the examples. An older version of the page shows + used in versions: https://docs.gradle.org/7.5.1/userguide/single_versions.html
Context (optional)
No response
Page with the problem
https://docs.gradle.org/8.7/userguide/single_versions.html

It's already fixed by https://github.com/gradle/gradle/pull/28780


Update location map generator, update locations with latest changes, update templates,update availability to include library name in front.

closes #1434
closes #1431
closes #1437
fixes the JS display for https://github.com/pulibrary/orangelight/issues/2499

When you merge this PR you will need to update the locations on bibdata-alma-staging.

enter the rails console in the VM and run LocationDataService.delete_existing_and_repopulate
For local development you will need to generate from templates since the templates have been updated. This happens when you run bundle exec rake servers:start.

I ran LocationDataService.delete_existing_and_repopulate.


Smart Dialog

Hello, I created a smart dialog for this package, and it's verry useful
Example of usage:
import samp from "samp-node-lib";
samp.OnPlayerConnect((player) => {
    player.ShowPlayerSmartDialog(samp.DIALOG_STYLE.MSGBOX, "Smart Dialog", "Smart Dialog Debug", "Button 1", "Button 0", (response) => {
      if(response.button) player.SendClientMessage(-1, "Clicked button 1");
      else if(!response.button) {
          response.repeatDialog();
          player.SendClientMessage(-1, "Clicked button 0 and repeated the dialog");
      }
    });
});

Nice job.


[GT-1930] Add the ANYPOINT_PLATFORM_CLIENT Credentials

https://jfc-dcd.atlassian.net/browse/GT-1930
What happened 👀
Pass the ANYPOINT_PLATFORM_CLIENT_ID and ANYPOINT_PLATFORM_CLIENT_SECRET so the app can have these Properties
Insight 📝
1/ This is not a required config, the other app can deploy without this information as long as they don't need it
2/ PR on The App https://github.com/Jollibee-Foods-Corporation/jfc-global-store-proc-api/pull/80
Proof Of Work 📹
On the change file tab

Just curious, what are those values used for?

Just curious, what are those values used for?

It's for the Autodiscovery in this PR @hoangmirs https://github.com/Jollibee-Foods-Corporation/jfc-global-store-proc-api/pull/80


Google sheet fetching is slow

What happens right now:
We use tabletop lib. to fetch data from google sheets which can be slow.
What I propose
Since our data don't change every minute or even hour, we can run some script to fetch and store data in json format in our repo only, Gatsby can fetch data from the json faster, browser might also cache that json (will hav to look into it) But switching from tabletop to json in our repo will definitely increase our speed of loading.
Steps:

make a script that query once the google sheet and save the required data in json format in our repo.
JSON gets saved in our github repo.
we commit the new data
redeploy our website.

What will the change means to others:
Right now I think simply adding data to google sheets works and those data immediately reflects in website. Whenever data is updated, we will have to redeploy the website. An extra step but it wont be too frequent.
We might use some hooks in future to automate the extra step too but will be a seperate feature.
@ifaisalalam @ujjwal8880  @rishukr06 @rohitbind

Do you guys approve my proposal ?

Ha , That will be too faster .
On Sat, 20 Jun 2020 at 12:51 AM, Mohit Gupta<EMAIL_ADDRESS>wrote:

Do you guys approve my proposal ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/thevaccinetracker/thevaccinetracker/issues/2#issuecomment-646826104,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJE2CJLJ6R4L5KESPKNCJP3RXO3D7ANCNFSM4OC7K6SQ
.

--
Regards,
Ujjwal Singhal
Software Development Engineer
Mobile: +91-9528288880
[image: logo] https://www.rentomojo.com/
Rentomojo, No.2, 26, 27, Sona Towers, 3rd Floor,
Krishna Nagar Industrial Area,
Hosur Main road, Bangalore-560029
www.rentomojo.com
--
The contents of this email message and any attachments are intended solely
for the addressee(s) and may contain confidential and/or privileged
information and may be legally protected from disclosure. If you are not
the intended recipient of this message or their agent, or if this message
has been addressed to you in error, please immediately alert the sender by
reply email and then delete this message and any attachments. If you are
not the intended recipient, you are hereby notified that any use,
dissemination, copying or storage of this message or its attachments is
strictly
prohibited.-------------------------------------------------------------------------------------------------------------------

Instead of updating JSON into the app repo, I propose a better way would be to store the JSON file separately, like in a S3 Bucket on AWS. We can write a Lambda function on AWS which will run periodically and update the JSON file in S3. Our Web and Mobile apps can fetch the data directly from this JSON file hosted on S3 bucket.
Considering the size of the file and the frequency of updates, the net bill should almost be zero dollars.

Cool I like that too.
On Sat, 20 Jun, 2020, 1:14 AM Faisal Alam<EMAIL_ADDRESS>wrote:

Instead of updating JSON into the app repo, I propose a better way would
be to store the JSON file separately, like in a S3 Bucket on AWS. We can
write a Lambda function on AWS which will run periodically and update the
JSON file in S3. Our Web and Mobile apps can fetch the data directly from
this JSON file hosted on S3 bucket.
Considering the size of the file and the frequency of updates, the net
bill should almost be zero dollars.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/thevaccinetracker/thevaccinetracker/issues/2#issuecomment-646834179,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFTAFNFQQD6X7KD4OIBFHKLRXO53BANCNFSM4OC7K6SQ
.

I think both the ideas are great. @ifaisalalam idea can help us setting up our future infrastructure.
For time being to cut down the API calls @shaileshrv has setup the session for now. Soon we can work on @ifaisalalam's idea.

@rohitbind @shaileshrv Doesn't session storage gets deleted when page ends. We have a single page app and no other page. Loading for the first time will still call the api. This session I am not sure will solve the problem of slow loading. This will only solve for those cases where user doesn't close the tab. and refreshes.
https://developer.mozilla.org/en-US/docs/Web/API/Window/sessionStorage read for more reference why session might not be the best way to solve it.
@shaileshrv have you tested and observed faster loads after using session ?

Sorry @thedeveloperr the session was added for some other feature. My bad this is obviously still open. It will be really great if you could pick this issue up.


Sdcard write throughput drops when used in conjunction with the HTTP Server (IDFGH-4957)

Environment

Development Kit: ESP32-DevKitC
Kit version: v4
Module or chip used:  ESP32-WROOM-32U
IDF version: v4.2
Build System:  idf.py
Compiler version: 8.4.0
Operating System:  Linux
Using an IDE?: No
Power Supply:  USB

Problem Description
Hi everybody,
I am using a small SDIO breakout board to connect a 8GB SDHC/SDXC card to my ESP32-DevKitC. While the write performance is pretty good (~3MB/s) when tested isolated, it significantly drops (~0.3MB/s) when the Sdcard is used in conjunction with the HTTP Server.
Please see the two example programs below.
Expected Behavior
For a Sdcard connected in 4bit SDIO mode, writing to a FAT filesystem I would expect  a throughput of ~3MB/s
Actual Behavior
Under certain circumstances the throughput drops to ~0.3MB/s
Steps to reproduce
Example program 1 (the fast one)
This example program here fills a 32k buffer 1000 times with As and writes it to the Sdcard.
That's the output of the program:
I (340) example: Initializing SD card
I (340) example: Using SDMMC peripheral
I (350) gpio: GPIO[13]| InputEn: 0| OutputEn: 1| OpenDrain: 0| Pullup: 0| Pulldown: 0| Intr:0
Name: SU08G
Type: SDHC/SDXC
Speed: 40 MHz
Size: 7580MB
I (560) example: Opening perf.txt for writing
I (10960) example: File written
I (10960) example: write time = 8106
I (10960) example: total time = 10293
I (10960) example: Card unmounted

Example program 2 (the slow one)
This example program reads data from a HTTP post request and writes the data to the Sdcard.
That's the output of the program:
I (2606) example: Starting server on port: '80'
I (2606) example: Registering URI handlers
I (4326) example: Initializing SD card
I (4326) example: Using SDMMC peripheral
I (4326) gpio: GPIO[13]| InputEn: 0| OutputEn: 1| OpenDrain: 0| Pullup: 0| Pulldown: 0| Intr:0
Name: SU08G
Type: SDHC/SDXC
Speed: 40 MHz
Size: 7580MB
I (129216) example: DONE
I (129216) example: write time = 76780
I (129216) example: total time = 124692

As it can be seen in the output, the write time here is 76 seconds, whereas the write time in the example program above is just 8 seconds.
Steps to reproduce

Connect a SDIO breakout board to the ESP32-DevKitC v4 (in 4bit SDIO mode)
run the first example program ("the fast one"). -> you should see a throughput of ~3MB
run the second example program ("the slow one") and post a file via curl (e.g: curl -X POST -F<EMAIL_ADDRESS>-H 'Expect:' <IP_ADDRESS>/upload) -> you should see a significant drop in throughput.

Additional Information

the same hardware was used in both tests
the same sdkconfig was used in both tests
the same Sdcard was used in both tests
the same amount of bytes were written to the Sdcard in both tests
the same buffer size (32K) was used in both tests

Does anyone have an idea what's going on here? I am stuck at this for weeks now and I am running out of ideas of what I could try to nail the problem. Am I doing something wrong here?
Thanks a lot,
Bernhard

Hi Bernhard, from a brief look, only a difference in buffer sizes is apparent.
In the first case you have a 32 kB buffer, which is large and multiple of the allocation_unit_size specified in FAT settings (16kB). This allows FAT to write directly to the card, bypassing internal buffers.
In the second case, the buffer is 500 bytes, which isn't aligned to the SD card sector size (512 bytes) and much less than the FAT cluster size. Writing in 500 byte blocks requires the FAT filesystem to do read-modify-write operation sector by sector, which is quite slow.
Please try changing the buffer size in the 2nd example to match the 1st example and see if the results become closer.

Hi Ivan,
thanks for the fast reply, very much appreciated!
I think the buffers are the same in both examples. I am collecting the small 500 byte chunk in a 32k buffer (totalBuf) and writing the 32k Buffer to the Sdcard (see https://gist.github.com/bbernhard/bba404e297e6e136d54a1eb1b58f2fcd#file-main-c-L128).
Or am I missing something here?

Thanks for pointing this out, I have overlooked the 2nd buffer!
It seems that the amount written from the larger buffer may be less than 32k, if the next small buffer would overflow the large one. Because of this, each subsequent buffer will not be aligned to cluster boundary, potentially requiring more time due to a read-modify-write. This shouldn't reduce performance as much as 10 times, though.
I have modified the 2nd example to fill data directly into the larger 32k buffer, since it is already allocated. The example also tries to write in exactly 32k chunks. The write speed, if we consider only the time of write operation, is now very similar between the two examples:
1st (writing 32768000 bytes, same code as yours, only added kB/sec logging):
I (8960) example: write time = 6342, kB/sec=5045.73
I (8960) example: total time = 8551, kB/sec=3742.25

2nd (writing 33554432 bytes, modified code here):
I (84609) example: write time = 6221, kB/sec=5393.73
I (84609) example: total time = 76202, kB/sec=440.33

Note that the total time in the 2nd case is still significantly lower. I expect that reading from socket needs to be happen in parallel with writing to the SD card, to reach higher speeds. You can send the data coming from the upload request into a ring buffer and run a separate task to move data from ring buffer to a file.

Hi Ivan,
you were right on the money with that!
Originally I was using two tasks (one task to receive the data and one task to write the data to the Sdcard), but when I did that, I experienced the same problems as above. The write speed was just terrible.
I always knew that I wasn't filling up the buffers up to the last byte, but I always thought that this was "good enough" and can't be the reason for the slow writing. I've never been more wrong (I've also read up a bit on how a Sdcard & the fat filesystem works and now it makes perfectly sense to me).
I rewrote my two task based application today and made sure that every buffer is full up to the last byte before writing it to the disk. With that I was able to improve the performance of my application by a factor of 4. Receiving + writing a 18MB file now takes ~13 seconds instead of ~55 seconds.
I guess I could probably still improve that a bit by playing with the buffer size of my fifo/ringbuffer and by some code improvements. But with that performance I am already REALLY happy.
Thank you so much @igrr for taking the time to help me fix this issue. Very much appreciated!

Good to know you got it working! I'm going to close this issue now.
If your code isn't proprietary, would you mind posting a link to a gist with the new, faster version? I'm sure this will help those who may stumble upon this issue report in the future.

If your code isn't proprietary, would you mind posting a link to a gist with the new, faster version? I'm sure this will help those who may stumble upon this issue report in the future.

Currently it's still closed source, but I am planning to change the license soon (just need to get some legal things in order). I'll add a link here once it's open source :)


fix: Filter dependencies are not being applied in default values

SUMMARY
Fixes a bug where the default values of a filter were not being filtered by their dependencies.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
https://user-images.githubusercontent.com/70410625/163246774-ba8d5561-b1a0-4655-a66d-0163320c0ba8.mov
TESTING INSTRUCTIONS
Check the video for instructions.
ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

/testenv up


K8s 1.29 support: replace cis-1.23 with new cis option in RKE2

From https://github.com/rancher/rancher/issues/43110 (and @snasovich) as Kubernetes 1.28 & 1.29 bumps are on the radar:

In 1.29, the cis-1.23 option is scheduled to be removed in favor of cis in RKE2 1.29 releases per https://github.com/rancher/rke2/blob/9e79849266360b1aa27fabe0d981bcc6bc069858/pkg/rke2/rke2.go#L274

Backend is looking into impact but the UI does have this setting in place.

Leaving as Backend Blocked for now until it's clear that UI can start work on this.

By all apperances, k8s 1.28 doesn't require any UI so this may just be a 1.29 ticket in the end.
To watch: https://github.com/rancher/rancher/issues/43110#issuecomment-1918808001 indicates that there may be some API breakage with continued PSP usage.

This will require a backport to 2.8.x once we know when the backend is ready.

We'll likely have to follow a similar pattern to 1.28 for updating the appropriate dropdowns to include newer versions: https://github.com/rancher/dashboard/issues/10504 for reference.

/backport v2.8.next2

Nothing is hardcoded in the UI from analysis. And there is a deprecation warning already in place from an earlier k8s bump.
Questions that arose that will require some research:

what is the expected behaviour for folks upgrading from versions that supported cis-1.23? We likely need to check for the deprecated (and soon-to-be removed) options and then provide indication that the option will go away.

The options come from /v1-rke2-release/releases <kube version, for example <v1.29.4+rke2r1> agentArgs.profile.options[1].

Not quite sure where we display it thought

This is display in the 'Basics' tab of RKE2 provisioning. We may already be checking for unsupported cis values but someone will need to verify: https://github.com/rancher/dashboard/blob/master/shell/edit/provisioning.cattle.io.cluster/tabs/Basics.vue#L213C5-L213C19

Nice work @momesgin!
@gaktive Can we transfer this to host busters?

No UI changes required. Putting in To Test so that QA is aware once k8s 1.29 is done, they can verify this appears as cis.

I have verified that if cis-1.23 option is removed, the UI shows a banner informing the user of the deprecation


sql: Replace similar tables with templated queries

https://github.com/open-contracting/kingfisher-summarize/issues/139

@jpmckinney This is only some of the suggested tables, but I thought it might be useful for you to review the approach before I proceed with the rest.

Also, calling EXECUTE all in one file means that this file will need to wait for tender, award, and contract dependencies to complete, which will make Summarize a little slower. Can we instead have the CREATE TABLE query in a function, and call the function from separate files, so that Summarize can schedule them more aggressively?


Consider changing the bit-order in a byte from right-to-left to left-to-right

RISCV should changing the bit-order in a byte ,to left-to-right ,from right-to-left .
Multi-byte order is already Little-endian ,but completely opposite for single-byte ,due to history
Example
-------------------------------------------
|  Value  |   Current     |   Proposal    |
-------------------------------------------
|     1   |   0000 0001   |   1000 0000   |
|    15   |   0000 1111   |   1111 0000   |
|   128   |   1000 0000   |   0000 0001   |
|   254   |   1111 1110   |   0111 1111   |
-------------------------------------------

History

The latter was decided to resemble Big-endian ,which was designed to resemble the decimal/hindu-indic number-system
Later Little-endian was found to be more suitable/natural for computing operations ,and allowing great flexibility through simple casting (to different sizes)

Why ?

This would simplify the programming for the platform ,by merging the single-byte and multi-byte order ,instead of diversion due to historical reasons
This would solve the only issue of Little-endian ,which is intuitivity ,while analyzing/manipulating raw-memory (although tooling can create artificial views ,but its has very-bad experience)
right-to-left is indeed sensible for decimal number-system ,because its actually read by humans ,but memory is completely hidden/abstracted from the programmer ,hence switching to a side fully is the correct approach ,instead of both-legs-in-separate-boats
This is crucial ,because otherwise we leave a place for improvement ,possibly digging own eventual grave

Isn't it a Breaking Change ?
Not necessarily ,because byte swapping (on load/store ? or as opcodes ?) is already under-consideration ,for Big-endian ,so why not make it definitive(complete inversion ,instead of per-byte/partial)
Its too-late now ?
so RISCV is outdated ,before even releasing fully ?
Bound to be replaced ,too soon .
Related

http://groups.google.com/a/groups.riscv.org/g/isa-dev/c/9tg4gGg8w-c/m/ADy-5lrHAgAJ

This is not a programmer-visible detail of any RISC-V machine. If you want to draw out your bitfield diagrams with the LSB on the left, or maybe even number the bits starting with 0 at the MSB like in an IBM manual, then you're free to do that.
I don't think RISC-V even stipulates that memory stores bytes with a binary representation; it just requires that register values survive a round-trip through memory, and states the correspondence between byte address and numerical significance (i.e. the endianness). You can think of memory as being base-256 rather than base-2, and then any confusion about bit order goes away.

Right, if memory is byte-addressed, this is a presentational detail that is not visible to software.  So if you prefer to think about it that way, that's fine, it's compatible with RISC-V.
As a separate matter, I think most people find the current presentation more intuitive, in large part because Arabic numerals in English are written with the MSB on the left.

This is not a programmer-visible detail of any RISC-V machine.

What about bit-shift(and other bit-wise) operations of programming languages ,aren't they exposing a big-endian order ,completely visible to the programmer ,with pragmatic access ?

I think most people find the current presentation more intuitive

How can inter-mixing be more intuitive ,than consistency ;its simply stockholm-syndrome

because Arabic numerals in English are written with the MSD on the left.

I acknowledge it in my post ,that it resembles decimal-number-system ,but by that logic ,shouldn't the endianness be Big ,instead of mixing 2  opposite endianness

@aswaterman
please reply ,and also if RISCV cannot be changed now ,what are the available approaches to present the order as fully little-endian ,in a programming lang. like c

Ratified arch extensions (including the base RV32 and RV64 ISAs) cannot be changed.  RISC-V natively is a little-endian architecture and is expected to remain so (although it does provide support for big-endian data accesses).
Past that, as @Wren6991 and @aswaterman noted in earlier posts, your issue is a presentational detail that is transparaent to software and ISA semantics (and hence not an architecture matter).  Hence, as far as who to take this issue up with, I'm not sure.

RISC-V natively is a little-endian

thanks for your useful reply .

RISC-V natively is a little-endian

this is what im stating ,that why settle for "partial little-endian" ,instead of full support ,for both single-byte(bits of a byte) as well as multiple bytes (as already supported)
thanks


Handsake fail on nginx 1.11.13 with OpenSSL_1_1_0-pre6-2220-gb3c42fc25

Hey,
Litte bug report , that took some time to trace down to ct,
I'm running a test server with TLSv1.3 and with the latest few post draft 19 off TLSv1.3 implementation of TLSv1.3 I get server handshake fail in all browsers tested,
However after I comment out ssl_ct in config,  site works again

ssl_ct on;
ssl_ct_static_scts /etc/nginx/ssl/sct;

nginx build options
nginx version: nginx/1.11.13
built by gcc 6.3.0 20170406 (Ubuntu 6.3.0-12ubuntu2)
built with OpenSSL 1.1.1-dev  xx XXX xxxx
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --pid-path=/usr/local/nginx/nginx.pid --with-pcre=../pcre-8.40 --with-zlib=../zlib-1.2.11 --with-http_ssl_module --with-openssl=../openssl --with-openssl-opt=enable-tls1_3 --with-http_gzip_static_module --with-http_addition_module --with-http_geoip_module --with-http_dav_module --with-http_stub_status_module --with-http_sub_module --with-http_ssl_module --with-stream --with-stream_ssl_module --with-mail=dynamic --with-http_v2_module --add-dynamic-module=/opt/nginx-ct --with-mail=dynamic
Is there away to get this working again with newest git of OpenSSL? or should one wait untill TLSv1.3 is final? (looks like draft 20 is coming out very soon)

FYI I'm seeing this too with nginx 1.13.0 and openssl HEAD from today.

I've been seeing this issue for a while atleast 2-3 weeks, at first I thought it was related this this(1).  In fact it might still be.
fyi, I also upgraded nginx to 1.13.0 now with OpenSSL_1_1_0-pre6-2248-g7531b3a6c issue  precisest

https://mta.openssl.org/pipermail/openssl-dev/2017-March/009146.html

It's a bug in OpenSSL, I've submitted a PR: https://github.com/openssl/openssl/pull/3310/commits/230a90d2a926ee40aecccfab7140d956b7de00bf

Great @grahamedgecombe  Thank you for quick response :)

Fix has been merged into OpenSSL's master branch, closing

Great work, thank you. I've tested and it works.


lack config file

Could you share the config file named "/home/lwy/IGibson2021/iGibson/gibson2/examples/configs/locobot_interactive_nav.yaml"? Thank you!

And the file episode_dir = '/home/lwy/IGibson2021/iGibson/gibson2/data/episodes_data/interactive_nav'  ?


Performance test APK with 1000 subscriptions

Description:
$subject

Found a significant delay from enforcer side while having 1000 application,  1000 subscription and 1000 appmaps. Need to improve how we store these data in enforcer side to rectify this. For now I close this issue.


0013 - Roman to Integer

Description of the Problem
Roman numerals are represented by seven different symbols: I, V, X, L, C, D and M.
Symbol       Value
I             1
V             5
X             10
L             50
C             100
D             500
M             1000
For example, 2 is written as II in Roman numeral, just two one's added together. 12 is written as XII, which is simply X + II. The number 27 is written as XXVII, which is XX + V + II.
Roman numerals are usually written largest to smallest from left to right. However, the numeral for four is not IIII. Instead, the number four is written as IV. Because the one is before the five we subtract it making four. The same principle applies to the number nine, which is written as IX. There are six instances where subtraction is used:
I can be placed before V (5) and X (10) to make 4 and 9.
X can be placed before L (50) and C (100) to make 40 and 90.
C can be placed before D (500) and M (1000) to make 400 and 900.
Given a roman numeral, convert it to an integer.
Link To The LeetCode Problem
LeetCode

May I work on this issue ?

Is this issue already merged ?


Support Dapr state management building block

Description
Currently, OpenFunction only supports Dapr binding and pubsub.
We should add support for the Dapr state management building block following:

https://docs.dapr.io/developing-applications/building-blocks/state-management/howto-get-save-state/#step-1-setup-a-state-store
https://docs.dapr.io/reference/components-reference/supported-state-stores/

Additional context
To support Dapr state building block, we need to:

[ ] Create a proposal
[ ] Add state field to the function CRD and generate relevant context.
[ ] Add get/save/delete/query method to each language's functions-framework including go, nodejs, python, java and dotnet
[ ] Add samples
[ ] Add docs to the website repo

@wrongerror @tpiperatgod @geffzhang @webup @kehuili @wanjunlei

@shaolei7788 we need a proposal like this https://github.com/OpenFunction/OpenFunction/blob/main/docs/proposals/202112-functions-framework-refactoring.md


Error executing a native build of GraalVM 22.3 in aws lambda

Fixed including the config files for native builds


How to change NativeAds ad layout using flutter only

We could change NativeAds style by nativeTemplateStyle: NativeTemplateStyle, but how could we change the layout of an ad using flutter code only?

Hi @winterdl
Currently, there are two layout types from the supported template: medium and small. And you can pass your preferred type via NativeTemplateStyle's templateType.
Your question is best suited for support channels like StackOverflow, so please ask there for further questions. If you are proposing a change, please use the appropriate new issue template and fill out the requested details so that we can address this issue properly.
Closing from here as this is more of a how-to question rather than a bug or feature request.


Recent and Similar Technical Scope

Question/Comment on FedRamp Dashboard RFQ
Name and affiliation
Patrick McConnell, Agile BPA Manager for SemanticBits
Section of RFQ documents
RFQ 7.B.iii.iii. Recent and Similar Technical Scope
PWS 4.0 REQUIREMENTS
Question/Comment
Please confirm the Similar Experience requirement:

iii. Recent and Similar Technical Scope: Recent and similar technical scope is past experience with building usable, responsive, consumer facing web site serving at least a million users, built within the past three (3) years.

It does not seem to relate to the work required for this task order identified in 4.0 REQUIREMENTS:

Contractor will work with agency partners and 18F Identity Management team to build integration (using Security Assertion Markup Language (SAML) or other appropriate technology) between agency web properties and 18F identity management product.

The Government finds this to be consistent. If you do not agree, please provide similar experience requirements that you feel would make them more consistent.


Switch default branch to main

This will also trigger a rerun of the tests following the update of https://github.com/manics/repo2docker/tree/abstractengine in https://github.com/jupyterhub/repo2docker/pull/848
Edit: Looks like an upstream podman change has broken the test setup

https://github.com/manics/repo2docker-podman/blob/225265a8c09733250eb1a21efae4c12c7bf35e57/.github/workflows/test.yml#L71-L77


Improve graphql susbcription implementation

The subscription/streaming implementation could use some work (see https://github.com/tipsy/javalin/issues/1000#issuecomment-877836592)

I'm working. Thanks @dariuszkuc for all information.

That's great to hear @7agustibm, I just wanted to split it up so it would be easier to handle.

@7agustibm should this issue be closed?

For me, yes.

For now I would leave it as it is. I don't know if it is a very required feature.

Doesn't seem that way. Let's close it now and open a new one if required. Thanks!


fix(transformer): remove an ast.copy from NullishCoalescingOperator transform

Remove one unnecessary ast.copy call from NullishCoalescingOperator transform (towards #3483).

#4913  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @overlookmotel and the rest of your teammates on  Graphite

Minor uncontroversial change, so merging without review.


Call awakeFromNib on super

If you leave out the call to awakeOnNib on super it will generate a warning in Xcode 8.

Looks like this will never be merged...