Add word-search exercise

I don't really like the example solution, but it works. Also I'm not sure if it is the right way to provide the class Point this way.
https://github.com/exercism/x-common/blob/master/exercises/word-search/description.md

I like providing the Point class. It could've been done with just having them return the coords as a tuple, but this way is more explicit.
LGTM 👍

Hey @behrtam, I've got two questions on this:

Since in #272 you were quite in favor of very minimal skeleton files, why did you provide such a (huge)  skeleton file here?
Aren't we limiting the user by forcing the class-structure? (I do not see yet the added value of the "Point" class/general class structure)

Well it's four months away, but I assume I did it that way because the test suit relies on the class Point. This is probably a unpleasant result of my example implementation which I normally write first.
Maybe if we would instead test with tuples that represent a point, we would not need a dedicated class for that, which also means we could reduce the skeleton file and the people solving this problem would have more freedom.

Ah ok that makes sense, thank you.
If its okay with you, I'll try to refactor it a bit and send a PR :smile:

Okay i created PR #420, please review.
Please note: I also changed the order of point tuples from coordinate-based (x,y) to a (at least to my mind) more intuitive (row,colum) notation. (If you prefer the coordinate-system, i'll update it).


Update "state" only if value comes from device

Description:
Related issue: fixes #23351
After a power outage I verified the energy value of a zwave device (Qubino Smart Meter) to jump from 30.7 to 23.1 and back to 30.7 causing the utility_meter to jump accordingly.
This is due to the fact that when zwave starts it read the first value from the configuration xml file and not from the network. The configuration xml file is saved only when hass.io graceful restart (or stop) appens. So in case of a power outage the values stored in the configuration file can be very old (3 days in my case). This cause a "jump" in the utility_meter value. The result were an energy counter that counted the last 3 days twice.
My solution is to ignore the initial value. I can recognize this by using the is_set property.

is_set returns True if the value has actually been set by a status message from the device, rather than simply being the default.

Ref:
https://github.com/OpenZWave/python-openzwave/blob/master/src-api/openzwave/value.py#L417
Checklist:

[x] The code change is tested and works locally.
[x] Local tests pass with tox.
[x] There is no commented out code in this PR.
[x] I have followed the [development checklist][dev-checklist]

With this change, on startup, what will the state of a battery-powered sensor be until it wakes up for the first time to report its value? Will the entity not exist, or have an unknown state, or what?

Hi @OverloadUT , thanks for asking.
The entity state will be Unknown at startup.
IMHO Unknown is better than a wrong value.

I'm unsure about what problem this solves. My understanding is that you template/add up energy values of one or multiple zwave sensors to get to your utility meter value. As the sensor value for energy is usually cumulative for zwave sensors, this value jumping should only for a very short time confuse the reading (say a blip when you graph values). It should never in fact change the sum of values once sensors get added up.
I would very much dislike this change if it means my zwave sensor state from battery powered devices does not get loaded - some of these devices only report in once every 6 hours unless otherwise prompted.

Why aren't we just restoring sensor states from the state history? I was under the impression that a lot of platforms are moving in that direction? I'm a bit uneducated here though, so take it with a grain of salt.
I DO think the root issue should be solved: If Hass uncleanly exits, Z-Wave sensors will boot up with INCORRECT readings (from days or even weeks ago.) One of the golden rules of Home Assistant should be that we never set an entity's state to something we know to be not accurate. Automations could trigger based on false data which is very bad.
I really don't like Unknown states, but it's better than wrong readings. But my original question remains: Can we restore from history if available?

That only works i you have a states history. Which I do not for all possible zwave sensors. Or even most. I have eg a door template binary sensor based off a numerical value from my fibaro door sensor 2 - that I store in history. I need the zwave state to restore so my template sensor will work again and show the door as closed/open based on the numerical value. Or to not just throw awkward errors.
It does not make sense to save everything zwave has to disk all the time but zwave should still be able to come up with full entities.

@olbjan Here's the reality: If you do not store it the recorder database, and Hass does not cleanly exit, then the correct most recent state does not exist in your filesystem. The question becomes what should we do given that fact? Restoring to whatever happens to be in the Z-Wave cache, which can be days or weeks old, is way worse than going to Unknown until you get a new update from your device.
The recorder exists partially for this purpose. I propose that the state restoration come from the recorder database, and falls back to Unknown if it's not in there. Then it's on you to decide if you want to have the recorder enabled for each sensor.
The Z-Wave cache is like an unreliable second database that doesn't have all of the development and thought put in to it that the Hass recorder does. I agree with not relying on it at all.

Looking into this the zwave cache will be used differently by ozw in version 1.6. I would suggest getting that into HA first and once that is done looking at what can or still should be done.
Quote:
Manager::WriteConfig and zwcfg_*.xml rename.
This function has been depreciated. This name was confusing, as it was actually just saving a cache of the Z-Wave Network to a file. OZW now automatically saves the cache as each node reaches QueryStage_Complete, and the file name has been updated to ozwcache_*.xml to indicate it a cache file, and not a configuration file.
https://github.com/OpenZWave/open-zwave/wiki/OpenZWave-1.6-Release-Notes

Correction for this discussion: state restoration does not rely on the recorder or state history. State restoration works with their own JSON storage.
ZHA does state restoration itself, for battery it restores anything <2hr old. For powered plugs it just queries it.

Hi @olbjan ,
as you stated before there will be some changes in OZW 1.6 :

OpenZWave 1.6 Release Notes
[...]
OZW now automatically saves the cache as each node reaches QueryStage_Complete
[...]
from: https://github.com/OpenZWave/open-zwave/wiki/OpenZWave-1.6-Release-Notes
Looking on my OZW log file I see that a node reach QueryStage_Complete only once at startup.
So the problem is still here.

I need to ask your help because I don't have a battery powered device to test. The question is:
Are you sure that a battery powered device isn't queried for values at startup?

[...]
During the startup phase, OZW can be busy querying the capabilities and latest values from nodes.
[...]
from: https://github.com/OpenZWave/open-zwave/wiki/FAQ

Could you please test this scenario?
With the Z-Wave door sensor:

Close the door
Wait Hass to update the door state to Closed
Stop Hass and wait a couple of minutes
Open the door
Start Hass and wait a couple of minutes
Chek if Hass report the door as Opened or Closed

Thanks.

@snowmaso This is pretty simple to answer actually. Most battery powered devices on zwave cannot be queried at all by the controller unless manually woken up by pushing a button first.
This is part of the zwave feature set of a node, listed in node information -> capabilities.
The list for my battery powered door sensors is: zwave_plus, beaming, routing
Whilst my mains powered switches have: listening, zwave_plus, beaming, routing
My smoke alarms have: routing, zwave_plus, frequent, beaming
The important differentiations are listening and frequent. Listening is always-on and can be queried (but that is a bad idea for bandwidth reasons) and frequent means it'll wake up often enough on its own to get messages like alarms on other smoke sensors.
I can do your test tomorrow.

Just did your test. Now 10 minutes in and the window I opened while zwave was down still reports as closed. My assumption is this will not change until either 6 hours pass or temperature changes .5C and gets reported, whichever comes first.

A day late jumping in on this conversation but here's my thoughts.
I support getting rid of restoring states from the OZW cache file, and using the Home Assistant state restoration process instead.
Ideally I think a setup similar to zha would be good, where powered nodes get queried on startup, and battery nodes are restored to their previous state if it was updated within a certain timeframe, otherwise they show up as unknown or unavailable. We could potentially leverage the zwave device_config settings in configuration.yaml to add an option to either restore previous state, or show as unavailable until the device reports in again so it could be configured on a per device or per entity basis.

I just wanna say one thing: querying on startup may be fine on very small zwave installations but on large ones that is a non-starter. Look at how long it already takes for the zwave network to be started gracefully! My office installation takes about 10-15 minutes to start - what you're proposing might make the network close to unusable with all nodes being queried at once.

@olbjan: "Now 10 minutes in and the window I opened while zwave was down still reports as closed."

IMHO This is a bug because even if Hass doesn't know the state of the door, it still report it as closed, and it's wrong. unknown is the correct answer from a system that doesn't know.
@cgarwood: State restoration is for sure a good option for those values that can change only througth Hass commands:

the target temperature of a generic_thermostat
the value of an input_boolean or an input_number
etc..

Only for these kind of integrations we can bet that the restored state is also the real state at startup.
But I also understand that due to limitation of a battery powered device the better choice is to let the user choice. As you suggested.

OZW author here - As mentioned above, is_set is the correct method to determine if the Value is "from the device" or not (ie, loaded from the cache).
As for how it all (should) work -

Mains powered devices should very quickly add their ValueID's, and then refresh those values with the current values from the device upon OZW startup.
Battery Devices will sit at CacheLoad with "minimal" ValueID's exposed (mostly what we call static values - things that change very infrequently or not at all). Once the device "wakes up" (via a button press or after the WakeUp Interval Expires on the device) will we start creating ValueID's for that device. As long as the device stays awake long enough, we should also refresh all the values on the device (but some devices go back to sleep before we have a chance to refresh. Some devices allow you to setup a timeout before the devices return to sleep mode, some dont, so its hit or miss).

The above is true for 1.4 and 1.6
So in summary - Checking if Manager::IsValueSet is the way to determine if the value has been set by the device or not (but be aware, a "Cached" valueID can return true if the application has Set a value on the device - eg, via SetValue but might not actually represent the state of that value on the device due to Sleeping devices etc).

We should close this PR. Caching is a pain to get right.
Removing caching without introducing our own caching strategy would lead to a reduced user experience. Open Z-Wave has access to more data and experience when it comes to Z-Wave and we should leave as much as we can to them when it comes to Z-Wave.

I dare say - in the context of the Ops issue, the proposed patch isn't wrong. Its making sure the value is updated. But HASS might want to implement that is_set check higher up in your z-wave stack.


Assistance with Loading and Executing real_data.zip.

Hello,
First of all, thank you for your fantastic work. I was able to successfully run demo.ipynb. However, I'm having trouble understanding how to load and execute it using the real_data.zip file that you have provided. Could you please guide me on how to proceed? (e.g. which file/command should I use?) I appreciate your work and thank you in advance for your help.
Best regards,

Thanks for your interest in our work!
Each folder in real_data.zip contains recording of 1 episode for 1 real world task. replay_buffer.zarr contains information about the robot actions and subgoal labels. You can read the file using this command:meta_data = zarr.open(file_path, 'r'). To access the robot end effector poses for example, you can do robot_eef_pose = meta_data['data/robot_eef_pose']. You may find this thread helpful too https://github.com/real-stanford/reflect/issues/3.
videos/0/0/color contains all the RGB images and videos/0/0/depth contains all the depth images recorded for the episode. You can read them using the following commands:
from imagecodecs import imread

rgb = imread(f'appleInFridge1/videos/0/0/color/{step_idx}.0.0.0')
depth = imread(f'appleInFridge1/videos/0/0/depth/{step_idx}.0.0')

You might need to do pip install imagecodecs. The RGB images and depth images are recorded 30Hz (30 frames per second).
Finally, audio.wav is the audio track recorded. Please let me know if you have more questions!
Best,
Zeyi


New json function similar to Hive's json_tuple

Hive's json_tuple allows extracting multiple keys from an input json string at once. This PR implements a similar functionality (until Presto supports table generating functions) with a new json_extract_multiple function. I didn't want to use the name "json_tuple" as in the future Presto may support table generating functions and then we can have a proper json_tuple function.

I think instead we want to implement partial casts for JSON, that would allow you to pick out the keys from the map. Once array/map are optimized a bit more, there should be virtually zero cost to retrieving the keys. Opened: https://github.com/facebook/presto/issues/2756


fix(facet): 序号列应早于列头添加到group

👀 PR includes

✨ Feature

[ ] New feature

🎨 Enhance

[ ] Code style optimization
[ ] Refactoring
[ ] Change the UI
[ ] Improve the performance
[ ] Type optimization

🐛 Bugfix

[x] Solve the issue and close #0

🔧 Chore

[ ] Test case
[ ] Docs / demos update
[ ] CI / workflow
[ ] Release version
[ ] Other ()

📝 Description
序号列后添加，遮住了角头
🖼 Screenshot

Before
After

🔗 Related issue link

🔍 Self Check before Merge

[x] Add or update relevant Docs.
[x] Add or update relevant Demos.
[x] Add or update relevant TypeScript definitions.

:tada: This PR is included in version @antv/s2-v1.7.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version @antv/s2-v1.7.0-alpha.5 :tada:
The release is available on:

npm package (@alpha dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Not clear steps

After step 2, the console shows 'Waiting for the USB device to appear'.
It should be stated that steps 3 and on should be taken while this is the case. The typical user waits for step 2 to complete(which will never happen) before it takes an action on steps 3,4 etc.

Document details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: e0afe59b-d18c-5a53-62c3-a64a425c960e
Version Independent ID: 12dae0a7-49c3-6dfe-d3c1-9179e7f69c31
Content: Update your Azure Percept DK over a USB connection
Content Source: articles/azure-percept/how-to-update-via-usb.md
Service: azure-percept
GitHub Login: @mimcco
Microsoft Alias: mimcco

@kgalic - the documentation has been updated to include a message that the user should proceed to the next steps while the "Waiting for the Device to appear..." message is displayed.
https://docs.microsoft.com/en-gb/azure/azure-percept/how-to-update-via-usb


Make the "more" dropdown button more visible

In the entity management view (and all other views) the "..." button to the right is kind of impossible to see unless you know it is there.
This needs to be made more visible. Make it an icon or something instead so you actually know it is there.

We've changed these styles long ago, @Zyles – a very belated thank you for pointing it out.


[Remote multigraph 1/5] Add new keys to meta.json.

[x] Forked repo is synced with upstream -> github shows no code delta outside of the desired.
https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork
[x] Tests are passing? https://github.com/microsoft/DeepGNN/blob/main/CONTRIBUTING.md#run-tests
[x] Changelog and documentation updated.
[x] PR is labeled using the label menu on the right side.

New Behavior
Add new keys necessary for multigraph support to load and manage graphs. Make them optional since normal graphs do not need.

Do we need to update parameters of MultiWorkersConverter or anything to accept these parameters or should we expect users to add them manually?
Any unit tests needed for this? It seems meta.json is not directly tested anywhere?

Do we need to update parameters of MultiWorkersConverter or anything to accept these parameters or should we expect users to add them manually?
Any unit tests needed for this? It seems meta.json is not directly tested anywhere?

You probably don't need to change meta.json at all. Graph serving is independent from conversion and it should have its own config.


Snark witness

@lazovicff in here when I use ctx.assign_advice() function it returns None value. If I put correct instances manually, Value::known(Fr::one()) for example, it still returns None. I couldn't find the reason behind it. What can cause it? It works when I use assign_from_constant() function.

I approached from another perspective and found a way to work it out. I took reader out from Value type and did the rest of the calculations as we did before. At the end I sent it to the function by wrapping Value::known()

I approached from another perspective and found a way to work it out. I took reader out from Value type and did the rest of the calculations as we did before. At the end I sent it to the function by wrapping Value::known()

I see. The Value type is making a lot of issues. I will look into this as well


Feat: Create CODE_OF_CONDUCT.md #855

Issue Solved: #855
Changes Made

Added CODE_OF_CONDUCT.md in the root directory.
Defined expectations for behavior, procedures for reporting issues, and consequences for violations.

Sample Screenshot

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Async attribute when use

Hi there,
The first, sorry for my English.
Examples bug report
In pages/index.js:
import React from 'react';
import Link from 'next/link';

class Index extends React.Component {
  render() {
    return(

	<Link href='/about'>
           Go to about
       </Link>

   )
  }
}
export default Index;

===================================
In pages/about.js
import React from 'react';
import Link from 'next/link';
import Head from 'next/head';

class About extends React.Component {
  render() {
	return(

		<Link href='/'>
                   go back
               </Link>

               <button  id='button'  type='button'>Click me</button>

	)
  }
}
export default About;

=============================================
In pages/about.js, i'm embed    within  component of Nextjs.
xxx.js content:
console.log('xxx');
document.getElementById('button').addEventListener('click', function () {
  console.log('clicked');
})

=============================================
Describe the bug
after that, in browser enter url localhost:port/about and then enable console i seen 'xxx' in console but is duplicated. So, when click to button element, console show two times 'clicked'.
When embeb xxx.js, if don't have async attribute, this does not happen but document.getElementById('button') is not found coz js loaded before element rendered.
I don't understand about async attribute in nextjs. If have "async" attribute, in Web Console -> Network tab, xxx.js loaded two times, else xxx.js loaded one time.
Please tell me why this bug appeared
Many thanks!
To Reproduce

Go to Browser
Enter url: localhost:port/about and hit Enter
Click in to "Click me" button
F12 find console tab
See error: duplicated text "clicked"

Expected behavior
Just have one text: "clicked" in Console tab when click to button.
System information

OS: macOS
Browser: safari
Version of Next.js: "^7.0.2"

 elements are remounted on page switches / hydration, meaning that if you include a


[iOS Bootcamp] Prework - Review

Prework completed, please review, thanks! /cc @codepathreview

Good work. This prework is a preview of our weekly project process. Generally, weekly projects take between 5-10 hours to complete the required features and an additional 5-10 hours to complete the optional features. In general, we've seen that the more hours you log, the quicker you improve your proficiency with iOS.
The purpose of this project was to begin to explore Xcode and to get a broad overview of iOS development using Swift. For example, in this project, we explored the following concepts:

Code styling in Swift. You can find some code styling guides here:

Ray Wenderlich Swift Style Guide
Github Swift Style Guide

Views are created in Storyboard, Interface Builder, or programatically, but they have the same goal: instantiate, initialize, and layout view objects. We use IBOutlets to give names to view objects, similar to giving unique ids to divs in HTML.
We registered for touch events, which can be done programatically or via IBActions.
We explored NSUserDefaults, one of the four persistence strategies in iOS.
View controllers have a set of methods that are called when it loads, appears, or disappears. These are called view controller lifecycle methods.
Do your views look good on iPhone 4, 5, and 6? Do your views look good on both iOS 6 vs iOS 7/8? We will cover in class how to use Auto Layout to robustly design your views for different screen sizes and OS versions.
After this assignment, you should understand the purpose of IBOutlets and IBActions as well as the basics of designing views and programmatically interacting with the views from the controller.

The next step is to schedule a brief phone conversation with the teaching team here. You might have to click to subsequent weeks to find an available slot. Admission is rolling, so the sooner you schedule, the better. Once you've selected a time, update the calendar event with a phone number to call you at.
During the phone calls, I'll ask you a few simple language questions that may include things like: strong vs weak properties, optionals, blocks/closures, and categories/extensions. I'll also ask you about why you want to learn iOS and what apps you're interested in building.
Feel free to make any improvements to the app before we chat, I'll check it one more time right before our call. I give priority to people that demonstrate that they are willing to put a lot of time into this class.


Create modules in pkg/util

What does this PR do?

This PR creates the following modules within pkg/util:

pkg/util/cache
pkg/util/retry
pkg/util/optional -> created by moving pkg/util/optional*

Renamed type Optional to Option, and related functions

pkg/util/stats_tracker -> created by moving pkg/util/stats_tracker*

Renamed type StatsTracker to Tracker, and related functions

Motivation

Refer to RFC
Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

Reviewer's Checklist

[ ] If known, an appropriate milestone has been selected; otherwise the Triage milestone is set.
[ ] Use the major_change label if your change either has a major impact on the code base, is impacting multiple teams or is changing important well-established internals of the Agent. This label will be use during QA to make sure each team pay extra attention to the changed behavior. For any customer facing change use a releasenote.
[ ] A release note has been added or the changelog/no-changelog label has been applied.
[ ] Changed code has automated tests for its functionality.
[ ] Adequate QA/testing plan information is provided if the qa/skip-qa label is not applied.
[ ] At least one team/.. label has been applied, indicating the team(s) that should QA this change.
[ ] If applicable, docs team has been notified or an issue has been opened on the documentation repo.
[ ] If applicable, the need-change/operator and need-change/helm labels have been applied.
[ ] If applicable, the k8s/<min-version> label, indicating the lowest Kubernetes version compatible with this feature.
[ ] If applicable, the config template has been updated.

hi @liustanley
the change looks good,
but could you explain why the pkg/util/retry package need its own go.mod? 🙇

Hi @clamoriniere, I think I needed it in a previous version of my changes but realized that it's no longer needed in my overall refactor. It's a simple module so I can keep it in this PR or remove it, which would you prefer?

hi @liustanley
the change looks good,
but could you explain why the pkg/util/retry package need its own go.mod? 🙇

Hi @clamoriniere, I think I needed it in a previous version of my changes but realized that it's no longer needed in my overall refactor. It's a simple module so I can keep it in this PR or remove it, which would you prefer?

thanks for the quick reply.
Please remove it to keep the PR and repository as simple as possible 👍

@clamoriniere we need pkg/util/retry to be its module . it is used pkg/util/winutil . We need to convert pkg/util/winutil to a module. Serializer requires pkg/config, which internally depends on pkg/secrets that depends on pkg/util/winutil.


Add support for Svelte

Language name
Svelte (a frontend compiler, similar to React / Vue)
URL of example repository
https://github.com/Lleweraf/supachat
https://svelte.dev/repl/hello-world
URL of syntax highlighting grammar
https://github.com/sveltejs/svelte
Most popular extensions

Detected language
.svelte is similar to an HTML file.
It can consist of a script, html and css part.

Linguist already has support for Svelte:
https://github.com/github/linguist/blob/3c3b037910006fc2f1a9bb34b2c4e9cde062206c/lib/linguist/languages.yml#L6086-L6095
The repo you linked to even shows it as the most used language in that repo.
Closing as there's nothing to implement. If you are experiencing problem with classification in your repo, please open a new discussion.

https://github.com/Lleweraf/supachat/blob/latest/src/components/ChatInput.svelte
The JS and Style part are not syntax highlighted.

It is:

[...]

I've confirmed it with https://screenshot.guru/ too.
If you're not seeing it, it's probably something specific to you and outside of Linguist's control.

That's interesting. Then this is a bug in the (android) mobile app.

Yup, that's definitely outside of Linguist's control. I recommend you open a support ticket using the "Contact GitHub" link at the bottom on any page on the GitHub site.

As before, this isn't within Linguist's control @Darkle. I've tested things further and it looks like this is a problem with the grammar and not GitHub. You'll need to raise an issue with the upsteam grammar.

There seems to be an updated linguist to switch from a forked version of svelte-atom to the latest version of the official version. The new version is partially rewritten to inject js/CSS into svelte files.
And currently, it seems like the injected part doesn't work. Specifically this line https://github.com/sveltejs/svelte-atom/blob/1b2dbd6e14681365988a6798599e18b7887a38fb/grammars/svelte.cson#L26
This file is ported from sveltejs/language-tools https://github.com/sveltejs/language-tools/blob/master/packages/svelte-vscode/syntaxes/svelte.tmLanguage.src.yaml. The injection already doesn't quite work in the atom so it has been partially rewritten to supported in the atom. Is there any difference between the implementation of the atom and GitHub backend?
We can try to debug the problem and see if we can find a solution on our end. But maybe we can revert to the old version? Or I have another fork version forked from before we rewrote to use injection here. Maybe that's also an option?

Is there any difference between the implementation of the atom and GitHub backend?

Yes. Linguist processes and "compiles" the grammar files using our grammar "compiler" and creates a single JSON file which is then ready by the syntax highlighter.
You can see the current result of this "compiler" in the source.svelte.json in the linguist-grammars.tar.gz tarball attached to all releases - I suspect you'll be able to spot the problems when you see the resulting file. This precisely the file the highlighter uses.

would it be possible to use Svelte's official VSCode grammar from here?: https://github.com/sveltejs/language-tools/blob/master/packages/svelte-vscode/syntaxes/svelte.tmLanguage.src.yaml
(especially considering that Atom is basically done out here?)
i don't really know anything about TextMate grammars, but I've been following this issue around several repos ever since I first noticed it, so I'm trying to help where i can 😅.

would it be possible to use Svelte's official VSCode grammar from here?: https://github.com/sveltejs/language-tools/blob/master/packages/svelte-vscode/syntaxes/svelte.tmLanguage.src.yaml

Yup, that should work too.

I'm working on updating the VS Code grammar to make it usable with linguist.


Allow logged-in users to confirm/approve/post submissions via UI in addition to emailed links

Description
When a Submission is "accepted", it usually lands in a state where further action is needed before it is "posted" and the corresponding Document is created. That action might be "confirmation" by current or previous authors or "approval" by an AD or perhaps others. The required action depends on characteristics of the submission - e.g., whether it's a "00" rev or not, whether it's a group or independent submission, etc.
Those actions are currently carried out by following a link sent by email to the people who can take that action. The link includes an access token that is used to authenticate the request. This mechanism allows, e.g., authors who do not have a Datatracker account to carry out the action without having to set one up.
That works when it works, but sometimes email is lost or misclassified as spam, leaving a submission in a state where admin intervention is needed. The "full access" submission status page does not currently help with this as it only allows an accepted submission to be canceled.
For users who do have an account, these actions should be made possible via buttons or other appropriate UI on the submission status page. E.g., if an accepted submission is awaiting author confirmation, a logged-in author should be able to "confirm" the submission and cause it to be "posted" if the submission is awaiting that action.
Note that this should not prevent the actions from being taken based on the existing URL access tokens rather than by logging in. Supporting participants who do not have a Datatracker account should continue.
Code of Conduct

[X] I agree to follow the IETF's Code of Conduct

I suspect this will require showing the "full access" status page based on logged-in identity rather than only via URL access token as well.

Navigating to open ones by way of a "my submissions" menu item would go a long way.


How well does it perform on manga?

どうも、日本人じゃないです。How well does it perform on manga compared to TesseractOCR?

Try it. Don't forget to get the resolution right.


Cancelling a PipelineRun that is already running when new one is triggered

The question is for pipeline cancelation: a PipelineRun triggered by EventListener (let's say it's testing a github PR). To cancel the previous PipelineRun (still running) when another PipelineRun of the same type was triggered, is there some native way using special parameters?
Use Case/ Expected Behaviour:

Developer pushes to PR (PipelineRun triggered through event Listener)
Developer again pushes to the same PR (previous PR PipelineRun should cancel in favour of the new one that is triggered through the new push)

/kind question
/kind feature

See https://github.com/tektoncd/community/blob/main/teps/0120-canceling-concurrent-pipelineruns.md and related issues for current work in progress on concurrency controls.

Thank you @AlanGreene for the information. As per the threads is it possible to use to experimental feature of concurrency for now? do I need to update my tekton-pipelines to the latest version for it?

You can check out the experimental concurrency and workflows projects. Bear in mind that they are currently experimental and may change in breaking ways.
Feedback is always welcome and can be added via issues / discussions, joining us on Slack, or at the Workflows working group meetings: https://github.com/tektoncd/community/blob/main/working-groups.md#workflows


Phantom on-change events when binding to numeric input via value$

Description
Mousing-over the up/down UI widgets of a <input type="number" value$="param"> element triggers on-change events in Chrome.
Live Demo
http://jsbin.com/wucuzoz/3/edit?html,console,output
Steps to Reproduce

Mouse over up/down UI widget.
Keep trying.

Expected Results
Nothingness.
Actual Results
Function bound to on-change event fires.
Browsers Affected

[X] Chrome
[ ] Firefox
[ ] Edge?
[ ] Safari 9
[ ] Safari 8?
[ ] IE 11?

Did not test Safari 8, Edge, nor IE.
Versions

Polymer: v1.2.4/master
webcomponents: v0.7.20

JSBin showing the same behavior without polymer: http://jsbin.com/xucarow/edit?html,console,output

Looks like the first time you interact with an input of type=number that had value set with setAttribute, it will fire a change event.

This seems like a Chrome bug, filed https://code.google.com/p/chromium/issues/detail?id=588008.

Fixed in Chrome with https://codereview.chromium.org/1719843002


[Bug] stg_netsuite2_transactions.status does not provide "display" status

Is there an existing issue for this?

[X] I have searched the existing issues

Describe the issue
The current package setup returns 'A', 'B', 'C', etc as transactions status (differently from NS1).
NS2 provides a dedicated function to calculate the display status: https://timdietrich.me/blog/netsuite-suiteql-transaction-statuses/
Relevant error log or model output
No response
Expected behavior
The "display" status should be 'Fully Billed', 'Open', etc
dbt Project configurations
basic config
Package versions
0.7.1
What database are you using dbt with?
snowflake
dbt Version
1.3.2
Additional Context
Solution 1: package should foresee the join with stg_netsuite2__transactionstatus on status and type
FROM {{ ref('stg_netsuite2__transactions') }} AS transactions
INNER JOIN {{ ref('stg_transaction_status') }} AS status_map
    ON transactions.status = status_map.id
        AND transactions.transaction_type = status_map.trantype

Solution 2: adding the conversion in the _tmp model
Solution 3: a macro to get the value from the transaction status and type:
{% macro get_status_display_value(tr_status, tr_type) %}
    SELECT name
    FROM {{ source('netsuite2', 'transactionstatus') }}
    WHERE id = {{ tr_status }}
    AND trantype = {{ tr_type }}
{% endmacro %}

Are you willing to open a PR to help address this issue?

[X] Yes.
[ ] Yes, but I will need assistance and will schedule time during our office hours for guidance
[ ] No.

Hi @ropalloni thanks for opening this issue and for adding valuable context in order to understand the status mapping within Netsuite2. I am going to adjust the bug report to instead me positioned as a feature request since this seems to fit more in line with adding additional functionality rather than fixing something that is broken.
That being said, I think this would be a great feature to add to the package and provide users with more insight into their Netsuite2 transaction data. The only thing to call out is that in my own local data I do not see the transactionstatus source table. If this is implemented, we will want to take into account that not all users may have this source table at the start. Additionally, we typically do not prefer performing joins in the staging models. I feel the join you suggested would be best served as an intermediate model in the dbt_netsuite package. However, in this package we would need to still add the transaction status staging model to then be brought downstream.
I noticed you are open to creating a PR. If you want to contribute this feature to the package, I would be happy to review and help to integrate it into this package and the downstream dbt_netsuite package. Let me know! In the meantime, I will explore adding the transactionstatus table to my local data and I will continue to explore this.

I'm working on a PR for this now. I'd be happy to help integrate the transactionstatus table into the downstream dbt_netsuite package as well. In case you weren't able to get the transactionstatus source table, I found that you need to have the "Set Up Company" permission in NetSuite. Seems weird, but that's NetSuite 🤷


experiment: rewrite page ID as list of child indices

#137  👈
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @rphmeier and the rest of your teammates on  Graphite

and later re-encode it when it's time to fetch a page from storage

That we don't re-encode the path is actually the primary motivation behind the change. We use the limbs as the key in RocksDB, not the fixed-width representation. The fixed-width representation is rarely needed.

We use the limbs as the key in RocksDB, not the fixed-width representation

but if you use length_dependent_encoding as key
let page1 = ROOT_PAGE_ID.child_page_id(0);
let page2 = page1.child_page_id(0);

would page1 and page2 be the same? probably I don't know the inner behavior of ArrayVec and instead they might be treated differently

Yes, exactly - they have representations that are based on the length (which RocksDB stores anyway)
Root -> []
root.child_page_id(0) -> [0]
root.child_page_id(0).child_page_id(0) -> [0, 0]
So a page ID is actually made by "unpacking" the sextets of the keypath

they have representations that are based on the length

Oh, okay! Great! I still don't know if its faster to create all limbs initially or to do it only on-demand (this would mean having a PageId that stores both KeyPath and limbs). It is easy to implement, so I will give it a try and upload the results.
However, there is only one missing piece for me now: why does the store module use page_id.length_dependent_encoding() to write a page, while to load a page, it uses page_id.encode()? This should not work because the encodings are different, but tests and workloads execute without any problems

Oh, I just changed 'encode' and 'length_dependent_encoding' alternatively in the write and load pages, and the test passes anyway. We have to work on our test suite

Ha. This is a bug, thanks for reviewing. Will push a fix! Yes, the tests pass because we never purge the cache.

I still don't know if its faster to create all limbs initially or to do it only on-demand (this would mean having a PageId that stores both KeyPath and limbs)

I don't either, to be honest. It's not our biggest issue right now - this PR isn't meant to "finalize" the implementation of the PageId but rather just focus on the length-dependent encoding. I could see our in-memory workload being faster with either one or the other approach, or it just not mattering at all in the scheme of things.
RocksDB is going to handle ordered data better than unordered, so I'm hoping that this changeset (and more to follow) will bring down the commit times for pages by quite a lot.

Merge activity

Apr 22, 10:08 AM EDT: @pepyakin started a stack merge that includes this pull request via Graphite.


[bug] drop/delete terribly slow

Bug report
Influx version: v1.5.1
OS version: Ubuntu Linux 17.04
Steps to reproduce:

Have a big database with > 20000 series
Try to execute drop series from /{{regexp_pattern}}/.
or try to execute drop measurement /{{regexp_pattern}}/.
or try to execute delete from /{{regexp_pattern}}/.

It will take > 30 minutes.
Expected behavior:

Well deleting data should take miliseconds?
Or there should be some special command modifier that will do it fast (I think this is becaus esome influx DB cleaning stuff). And there should be another comment to execute this "long" part manually, so for example I can delete ton of series and run cleanup process manually (to reclaim space) from a cron job or something like this.

It definitely shouldn't be taking that long.
Can you gather some profiles while the slow delete is in progress?
curl -o profiles.tar.gz "http://localhost:8086/debug/pprof/all?cpu=true"

curl -o vars.txt "http://localhost:8086/debug/vars"
iostat -xd 1 30 > iostat.txt

The workaround I would try is to break out the regular expression, like running SHOW MEASUREMENTS WITH MEASUREMENT =~ /pattern/, and then make individual DROP calls in one statement, like
DROP MEASUREMENT match1; DROP MEASUREMENT match2; DROP MEASUREMENT match3....
If you run the DROP calls as separate statements, you may trigger a compaction on each run.

I will try to provide the details You need tomorrow.
But dropping series one by one is a lot faster, but still dropping any single series takes tens of seconds.

Uploaded requested profiles:
profiles.tar.gz
iostat.txt
vars.txt

Delete still running (already 19 minutes) it was at least 40x faster in v1.3.x.
Requested stats after about 17minutes running delete:
iostat.txt
profiles.tar.gz
vars.txt

Can I somehow turn compaction off?

I don't believe that is supported. Maybe setting max-concurrent-compactions=1 would suffice here. https://docs.influxdata.com/influxdb/v1.5/administration/config/#max-concurrent-compactions-0

Adding a point will either add new point or overwrite existing one.

Overwriting existing points is expensive and should generally be avoided if possible; otherwise it should happen in bulk if possible.

I also have a problem with influx db copy program (that I've. written myself months ago).

I suspect you'll get much better performance with using the native backup and restore tools.

Is there a way to tell influxd to allow unlimited number of connections or say at least 100? What is the default?

Default should be unlimited: https://docs.influxdata.com/influxdb/v1.5/administration/config/#max-connection-limit-0

Uploaded requested profiles:

Thanks, this will be very helpful in identifying the performance bottleneck.
@rbetts care to prioritize/assign this?

@lukaszgryglicki Thanks for the very complete report. We'll see triage this in an upcoming grooming session.

BTW: I've killed the query after 40 minutes and implemented alternate solution:

Program that zero-fills all series which match given regexp:

https://github.com/cncf/devstats/blob/master/cmd/z2influx/z2influx.go

I suspect you'll get much better performance with using the native backup and restore tools.:
BTW: is there a native program to copy one influx DB into another, or rename?
There wasn't such a program 3 months ago, I'm quite sure, because I was researching this for more than week some time ago?
Overwriting existing points is expensive and should generally be avoided if possible; otherwise it should happen in bulk if possible.:
All my programs use bulk writing.

BTW: is there a native program to copy one influx DB into another, or rename?

https://docs.influxdata.com/influxdb/v1.5/administration/backup_and_restore/

I'll duouble check tomorrow then but fast question:
Can I backup database named A into some file and then restore into database named B?
What I need to do

Assume I have to regenerate database A (calculations are slow, takes hours, data is generated from complex set of postgres SQLs)
I'm generating new influx datamase A_temp
If success then I want to rename A_temp to A (overwriting original A).
If there is an error with A_temp generation, A remains intact and A_temp (broken) is dropped.

Copy A_temp to A should be orders of magnitude faster than just generating A (to avoid downtime).

Seems like regexps are also very slow.
There is only one series name which match /^prs_labelled_d$/ and it is named "prs_labelled_d".
When I do:

select last(*) from "prs_labelled_d" it returns data instantly.
When I do:
select last(*) from /^prs_labelled_d$/ it takes 2m30s.

We've replaced influx with PostgreSQL and delete is instant.
Postgres (which is a surprise for me) is faster not only in delete operation, but in all operations basically.

This is kind of drifting off-topic, but how do you deal with the dynamic schema benefits of influxdb? Do you just not index the tags and store all measurements as strings? We have a fairly complex pile of pre-existing metrics and developing a sql schema for all of them in a reasonable way would be very difficult (unless we just crudely stored them as txt or something); the benefit of influxdb (and other storage systems like it) is that it is specific to the problem of metrics storage and querying (as opposed to general purpose databasing).

I can reproduce this bug, but not consistently. Created a database with 100000 measurements and about 8 million series. I have twice seen delete from /regex/ behave as though it were completely locked up.
Delete from 1 measurement:
$ time influx -database stress -execute "delete from m99999"

real    0m0.084s
user    0m0.020s
sys     0m0.022s

Delete from about 9 measurements:
$ time influx -database stress -execute "delete from /m9999./"

real    0m3.773s
user    0m0.024s
sys     0m0.022s

Delete from about 90 measurements:
$ time influx -database stress -execute "delete from /m999.*/"

real    0m3.106s
user    0m0.019s
sys     0m0.018s

Delete from about 900 measurements:
$ time influx -database stress -execute "delete from /m99.*/"
^C

Adding some debug logging to tsdb/store.go I can see that there is a deadlock. This also causes shutdown to timeout, and failover to a hard shutdown.

The deadlock is on partition.go, Partition.Wait() which simply calls i.wg.Wait().

Opening as there is still possibly a performance issue.

Related to #10056

master
$ time influx -database stress -execute 'delete from /m9.*/'

real    0m31.381s
user    0m0.021s
sys     0m0.021s

1.6
$ time influx -database stress -execute 'delete from /m9.*/'

real    0m29.174s
user    0m0.023s
sys     0m0.033s

1.5 (redo with dep ensure)
$ time influx -database stress -execute 'delete from /m9.*/'

real    0m20.289s
user    0m0.023s
sys     0m0.031s

1.4 (inmem)
$ time influx -database stress -execute 'delete from /m9.*/'

real    0m0.385s
user    0m0.023s
sys     0m0.030s

1.3 (inmem)
$ time influx -database stress -execute 'delete from /m8.*/'

real    0m0.413s
user    0m0.020s
sys     0m0.023s

@jacobmarble do you have an update about this investigation/bug fix ?

don't close.

I am using 1.7.7 and the issue still persists.
Deleting a single measurement with 1mio series takes forever.
iostat shows disk is not of a very high utilization
Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util loop0            0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00 nvme0n1          0.00  723.00      0.00   3046.50     0.00   163.00   0.00  18.40    0.00    1.62   0.07     0.00     4.21   0.05   3.60 nvme1n1          0.00  723.00      0.00   3046.50     0.00   163.00   0.00  18.40    0.00    1.84   0.10     0.00     4.21   0.07   5.20 md1              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00 md2              0.00  761.00      0.00   3044.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     4.00   0.00   0.00 md0              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00

Still present in 1.7.8.
10_000 measurements with delete query delete where time < '2018-01-01' took 25 seconds.

It's not just the delete that is slow, but the garbage collection and/or compaction that happens after the delete is terrible. I deleted a couple hundred measurements and the load jumped 6x for . Memory maxed. IO wait is 50% or so of all CPU time (cloud resources!).

If I remember correctly, we decided in 1.5 to block on delete, rather than accept the delete request and handle it asynchronously. My earlier analysis probably reflects this. So "that's a feature, not a bug" is my offhand comment for that.
@bryanspears have you tried limiting concurrent compactions? Where your bulk delete operation likely touched several shards, and those shards all require multiple levels of compaction, limiting concurrency has helped other folks in the past. Start with limiting to 1. https://docs.influxdata.com/influxdb/v1.7/administration/config/#max-concurrent-compactions-0

Possibly related #15271

@jacobmarble your suggestion has stabilized our small influx setup. For anyone else using limited I/O, cloud-based resources. These configuration options helped dramatically. Compaction does take longer, but that's better than the alternative crash that was occurring for us.
  wal-fsync-delay = "50ms"
  compact-throughput = "10m"
  compact-throughput-burst = "100m"```

@bryanspears I'm glad to know that limiting concurrent compactions helped. In many situations, compaction duration doesn't cost anything, and slower compactions free resources for queries and writes. For anyone else reading, when dealing with this sort of write-vs-read contention, I suggest

first set max-concurrent-compactions to 2
then set max-concurrent-compactions to 1
then set compact-throughput and compact-throughput-burst to values smaller than 48m
then wal-fsync-delay to 10ms

Be aware that setting wal-fsync-delay is often helpful for constrained I/O situations (fewer calls to fsync()), but it also risks losing the most recently written data (up to 50ms in the example) when power is lost. For many use cases, this is acceptable.
Another way to help WAL I/O is to write in batches, say 5000 points per batch. This way, you can let it fsync() with every call to the /write endpoint (wal-fsync-delay=0), since there are fewer such calls.

This issue was the first hit for a search about slow drop database operations. I'm on 1.7 and witnessed the same performance. I thought my network connection had dropped so after an hour I sent ctrl + c and it returned to the influx shell. So not the network. The database I wish to drop shows up in show databases but when I use it and show measurements I get the following output ERR: engine is closed.
I'm optimizing for disk space, so I very much would like to delete this temporary database.

I am cleaning up some invalid values from my measurement in python code. Table has data for a year with probably one minute resolution.
DELETE FROM Cool_Return_Temp WHERE time = $time;

Query is tremendously slow. i am thinking if it is possible to pass an array of times to make index rebuild only once.
Unfortunately next query was not working last time I tried.
DELETE FROM Cool_Return_Temp WHERE value > 50;

Experiencing the same difficulties here with InfluxDB v1.8, it takes more than 25min to drop 1 measurement where data is stored over 3 shards of 4 weeks duration. I have 42k measurements to delete it's not acceptable for our production platform, i'm gonna need to find a trick.

Wow, over 3 years passed since I've reported this and it's not fixed...

Not sure how related, but I've got a measurement with a single point on influxdb 1.8.5 and when I run DROP MEASUREMENT <name>, the operation just hangs forever. Not sure how to further debug. Rebooting influxdb does not seem to help

I am also getting the same issue, even I have only 7-8 measurements and none of measurement have more than 1500 records in it.
version:
CLI: Influx CLI 2.0.6 (git: 4db98b4c9a) build_date: 2021-04-29T16:48:12Z
DB:Version 2.0.5 ('7c3ead)
Delete command : curl -X POST
'http://<IP_ADDRESS>:8086/api/v2/delete?bucket=intralogistics&org=greyorange'
-H 'authorization: Token hzCzxEPyoXUcnJOkGQDuYzlMIQZ06jUKP8De-HKSQU5hmgXAAWevPXuZcrduV9OTIl-8phT2-ZFyd-sqMOeLSA=='
-H 'content-type: application/json'
-d '{
"start":"2020-06-20T00:00:00.000000Z",
"stop":"2021-06-20T23:59:59.999999Z",
"predicate": "_measurement="service_request" AND external_service_request_id="order_234_20210618135243""
}'
iostat.txt
vars.txt
profiles.tar.gz
I am at the default configuration so please let me know if I need to do set some configurations

We also want to drop measurements of hosts which do not exist anymore on influxdb v1.8.5. But after dropping a few of them the database just locks up and does not accept further drop statements. Then we have to wait up to a day until it is available again.
Would be nice to get this finally solved.
Also does someone have a clue if this issue persists in version 2.x?

experiencing this issue as well. we had a problem a while back which caused a measurement to be created with about 300k series. the new measurement is small, only about 1m rows, but trying to drop the measurement hangs forever. the cardinality of the database is blown up now (before this measurement, it was around 20k) and causing all kinds of problems.
i'm not even able to successfully drop any series from this measurement now. for example, a drop series query filtering on just 10 series hangs forever. nothing useful in the logs. any idea here?

Having this issue too, influx 1.8.10. In my case I'm using a query like:
DROP SERIES from /.*/ WHERE tag=test

I am still getting this issue.  DROP MEASUREMENT <name> is getting stuck and failing. When running from CLI, then getting 504 from the server after some time.
Sometimes when I delete some data from measurement (not drop) for some time period, It is not deleting and getting request timeout, no matter how much I increase the timeout. But when I restart the server then it gets deleted.

@bryanspears I'm glad to know that limiting concurrent compactions helped. In many situations, compaction duration doesn't cost anything, and slower compactions free resources for queries and writes. For anyone else reading, when dealing with this sort of write-vs-read contention, I suggest

first set max-concurrent-compactions to 2
then set max-concurrent-compactions to 1
then set compact-throughput and compact-throughput-burst to values smaller than 48m
then set wal-fsync-delay to 10ms

well years later I also suffer from this issue.
On Influx 1.8 I need to DROP 500k MEASUREMENTS. As this runs like 1 per 30seconds this is crazy slow and makes it impossible.
Is there no way to just drop them and when everything is dropped do indexing, compacting etc?
I did this little script to delete it .. but it needs to be A LOT faster to actually make sense
influx -username user -password pass -database homeassistant -execute 'SHOW MEASUREMENTS' | grep 'geo_location.lightn' | while read measurement; do influx -username user -password pass -database homeassistant -execute "DROP MEASUREMENT \"$measurement\""; done


Location not updating

I recently changed my location from Reykjavík to Paris. However I still see myself ranked in Reykjavík and Iceland. Is it gonna refresh soon or is it not supported?

GitHub Awards updates hourly based on data from GitHub Archive
You can force a refresh for your profile with the refresh button under your avatar.
I'm closing this issue, let me know if you have troubles updating your profile.

works :+1: I didn't know about that refresh profile thing. Keep up the good work :)


VLAN alloc with new KO

this PR implements the VLAN Allocation library with the new KubeObject aligned with SetSpec/SetStatus

/approve

I understand now, that the main concern is the lack of static type checking in the interface{} type parameter of SetSpec and SetStatus. I agree that this is a valid concern, however I suggest to use the power of Go generics to fix this, as opposed to implementing it one-by-one for each type (avoiding this is the main reason for having generics).
Here s my proposal to add static type checking: https://github.com/nephio-project/nephio/pull/141

/lgtm


Template Require via environment variable

Hey,
based on this Example (https://github.com/ampedandwired/html-webpack-plugin/tree/master/examples/custom-template) I created an similar setup which requires the sub html file via an environment var.
function generate_page(config) {
	var tempateUrl = config.templateUrl ? config.templateUrl : 'src/template.html';
	return new HtmlWebpackPlugin({
		filename:  config.filename,
		template: tempateUrl,
		xhtml: true,
		environment: {templateUrl:  config. subTemplateUrl}
	});

}

My src/template.html' requires this subtemplate:
<!doctype html>
<html lang="en">
<head>
    <meta charset="UTF-8">
</head>
<body>
<% require('html!'+ htmlWebpackPlugin.options.environment.templateUrl ) %>
</body>
</html>

But when I run  webpack  --progress --colors --watch -d I just get that error:
ERROR in Template execution failed: Error: Cannot find module "."

ERROR in   Error: Cannot find module "."

  - template.html:57 webpackMissingModule
    /Users/xxx/PycharmProjects/webpack-test/src/template.html:57:47

  - template.html:57
    /Users/xxx/PycharmProjects/webpack-test/src/template.html:57:125

  - template.html:62 module.exports
    /Users/xxx/PycharmProjects/webpack-test/src/template.html:62:4

  - index.js:228
    [webpack-test]/[html-webpack-plugin]/index.js:228:16

  - util.js:16 tryCatcher
    [webpack-test]/[bluebird]/js/release/util.js:16:23

  - promise.js:503 Promise._settlePromiseFromHandler
    [webpack-test]/[bluebird]/js/release/promise.js:503:31

  - promise.js:560 Promise._settlePromise
    [webpack-test]/[bluebird]/js/release/promise.js:560:18

  - promise.js:597 Promise._settlePromiseCtx
    [webpack-test]/[bluebird]/js/release/promise.js:597:10

  - async.js:131 Async._drainQueue
    [webpack-test]/[bluebird]/js/release/async.js:131:12

  - async.js:136 Async._drainQueues
    [webpack-test]/[bluebird]/js/release/async.js:136:10

If I write the same subTemplateUrl in the template.html file instead of the variable everything works fine:
<% require('html!./test.html' ) %>

It seems like your variable contains only ".".

Yes, but this is not true.
If I write the variable instead of require it than I get the correct url.
<% htmlWebpackPlugin.options.environment.templateUrl %>
-->
./test.html

Oh - I believe webpack can't handle dynamic require statements.

Basically it should:
https://webpack.github.io/docs/context.html

But your are using it wrong.
Please ask questions about the webpack context require statements on stackoverflow.

Hi Jantimon,
thank you for your response. I added this question to stackoverflow.
After you figured out that I'm using it wrong I'm sure that you know what I do wrong and answer the Question at Stackoverflow:
http://stackoverflow.com/questions/36387544/require-variable-with-html-webpack-plugin
:-)


Support for redirect to webpage when signing up

We are implementing redirecting user to a webpage when signing up, so recaptcha, or maybe google login in the future could be used
The command sent by the server is REGISTRATIONURL https://something.com/abc.php
for example, when trying to signup, will be live soon in lobby.techa-rts.com
I tried to do it , but can't get it to compile at all

https://github.com/tizbac/skylobby/commit/8b777d385c0866f2e0815f4f4d4b18627c6084f6 feel free to import if i didn't wreak too much havoc :D considering my clojure skill


object ids in tracked_objects skipped a number?

Hi,
I am using norfair to track faces. I noticed that sometimes tracked_objects were skipping a number for object ids. For example, I have 3 faces and object ids are 1, 3, 4 instead of 1,2,3. What could be the reason? Might I be doing something wrong? Let me know if you need anything additional. Thanks!

This is the video you can use for testing:
https://github.com/tryolabs/norfair/assets/140737044/aedd01ff-e2e3-458f-b8ba-c7be153cc038
Please refer to issue #307 for code. The only changes I have made are:

skip_period = 3,
initialization_delay = 9
hit_counter_max = 15

Thanks

@aguscas Could you please advise on this?

Alright, I made a test using the same parameters you said for all variables, but couldn't reproduce the problem you were mentioning about skipping ids. I'm still not sure what might cause you that. Here is the video created with your code.
By the way, these models run so slow that I shouldn't even call it 'running', it's more like crawling towards the output. Hope you find other models that run faster.
Also on another note, parameters seem to need a lot of tuning, lots of ID switches there.

I should have mentioned it before. It doesn't happen for every video. I have seen it happening when you have more faces (5+) in the video. I'll try to find an example video and share with you.

Here's one video you can try. You'll notice that object ids 26 and 29 are skipped:
/home/anurag/Work/FaceRedaction/test_videos/London Walk from Oxford Street to Carnaby Street_30secs.mp4

Are you sure you haven't made any other changes to the code here apart from the values of skip_period, initialization_delay and hit_counter_max? I am asking because I just tried that video with your code and couldn't even get to ids that high. Are you sure you haven't at least changed any threshold, like the distance_threshold or any other tracking parameter?

Closing this issue. I just added a postprocessing step to assign contiguous object IDs. Thanks!


Fix pod task fast registration template substitution

Signed-off-by: Katrina Rogan<EMAIL_ADDRESS>TL;DR
Fix pod task fast registration template substitution by examining container args in k8s_pod task template target.
Type

[x] Bug Fix
[ ] Feature
[ ] Plugin

Are all requirements met?

[x] Code completed
[ ] Smoke tested
[x] Unit tests added
[ ] Code documentation added
[ ] Any pending items have an associated Issue

Complete description
How did you fix the bug, make the feature etc. Link to any design docs etc
Tracking Issue
https://github.com/flyteorg/flyte/issues/1531
Follow-up issue
NA

Want to fix it there?
On Tue, Sep 28, 2021 at 6:17 PM Katrina Rogan @.***>
wrote:

@.**** commented on this pull request.
In cmd/register/register_util.go
https://github.com/flyteorg/flytectl/pull/184#discussion_r718073573:

@@ -211,6 +214,40 @@ func hydrateIdentifier(identifier *core.Identifier) {
}
}

+// TODO: once this code is moved to flytestdlib, refactor to remove this duplicate method.
+func marshalObjToStruct(input interface{}) (*structpb.Struct, error) {
actually this doesn't work because casted returns true here (incorrectly)
https://github.com/flyteorg/flytestdlib/blob/master/utils/marshal_utils.go#L66
—
You are receiving this because your review was requested.
Reply to this email directly, view it on GitHub
https://github.com/flyteorg/flytectl/pull/184#discussion_r718073573, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAGUFZK3D35A4CLIOBPGMDUEJSJPANCNFSM5E6QHR4Q
.

--
Thanks,
Haytham Abuelfutuh.

PTAL @pmahindrakar-oss @EngHabu


加载html 乱码

Navigate 中加载的html 具有中文 会出现乱码，

Please ask your question in English if you need help quickly. Thanks.

@zserge can this be closed?


Add API diff between .NET Core 2.2 and .NET Core 3.0 preview1.

Generated from the following two commits (including System namespaces only):
Oct 30 (.NET Core 2.2) - https://github.com/dotnet/corefx/commit/694e15f9974011662b3d5d17b2e3c248b33b527e
Nov 21 (.NET Core 3.0 preview1) - https://github.com/dotnet/corefx/commit/57358b7f7214c233e8397483a9afc1698ac00c5b
I only included what's in-box based on previous API diffs.
cc @richlander, @terrajobst, @leecow
Please do a sanity check for your feature areas.
cc @stephentoub, @tannergooding, @karelz, @JeremyKuhne, @steveharter, @GrabYourPitchforks, @tarekgh, @maryamariyan, @bartonjs

cc @terrajobst


GEOSgcm MAPL3 failure (2023-Jul-28)

I have been trying to trace these two issues
#2263
and
#2262
and have come to the conclusion they are really the same symptom.
The issue is in the MAPL_TerminateImportAllBut which was now being call in the surface component. The issue is this loop:
https://github.com/GEOS-ESM/MAPL/blob/release/MAPL-v3/generic/MAPL_Generic.F90#L5114
With gcc we get the crash references in issue #2262, with intel we get the crash referenced in #2263, but that is only because rather than failing the line referenced, the size(...) returns 0, and just skips the loop for all 4 children, so all their imports make their way to ExtData.
The fundamental problem is that the size( ) should not be zero, in fact for child 1 which is ocean, on develop I see that it is 47!
 It is almost like the SetService has never happened as that size( ) returns 0 for all 4 children of surface.

@atrayano has confirmed the MAPL release/MAPL-v3 branch now executes the SetServices in a different order so at this point the SetServices of the children not run so, this code in TerminateAllBut will simply not work on this branch.

Ugh.  That is unfortunate.
@mathomp4 Maybe we stop testing the MAPL3 branch for now.  After all for many months my development has been on features that are not yet used.   I jumped the gut with my early work.

Closed by #2270


Staging: General improvements to gateway

Before submitting this PR, please make sure:

[ ] Your code builds clean without any errors or warnings
[ ] You are using approved title ("feat/", "fix/", "docs/", "refactor/")

Description of the changes proposed in the pull request:

Make amm_arb handle MarketOrderFailureEvent properly
Update GatewayInFlightOrder to use InFlightOrder
Include nonce expiry for each nonce tracked and nonce architecture in general
Improve ConfigManagerV2 error messages
Add more details about gateway status in hummingbot
Use autorouter for uniswap
improve output from gateway status
avoid logging /network/status calls
simplify gateway create logic when infura api key is added
make initial docker not found error less technical and scary.

Tests performed by the developer:
Tips for QA testing:
Try running hummingbot and gateway with various amm_arb strategies and exchanges.

Closing this PR as it is already present in the latest release.


Memory Fingerprinting

From: https://github.com/ipfs/dynamic-data-and-capabilities/issues/50
Analyze the memory footprint of the log instance and make it smaller or configurable. There are scenarios where it's ok to have more I/O as opposed to increased memory usage. One example is precisely viewing the history of changes, which is a feature used sporadically.
See also #136

cc @satazor

@aphelionz You have a small typo in the title: fingerprinting -> footprint

See #136 as well for some work towards this.


Update to new version then legend didn't work

I Update my js from 3.6.12 to  3.10.0 then the legend didn't work
It happens only when using IE
like this :
3.10.0

3.6.12

code like this

is anything need to set?

Can you please paste your whole chart configuration?
Or better create a codepen demo?

I just post my hole code, codepen isn’t work on IE browser,thank for help


[instrumentation_adapter] Update README about migrating to AndroidX

Description
A tiny doc update.
Related Issues
N/A
Checklist
Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]). This will ensure a smooth and quick review process.

[x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[x] My PR includes unit or integration tests for all changed/updated/fixed behaviors (See Contributor Guide).
[x] All existing and new tests are passing.
[x] I updated/added relevant documentation (doc comments with ///).
[x] The analyzer (flutter analyze) does not report any problems on my PR.
[x] I read and followed the Flutter Style Guide.
[x] The title of the PR starts with the name of the plugin surrounded by square brackets, e.g. [shared_preferences]
[x] I updated pubspec.yaml with an appropriate new version according to the pub versioning philosophy.
[x] I updated CHANGELOG.md to add a description of the change.
[x] I signed the CLA.
[x] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Does your PR require plugin users to manually update their apps to accommodate your change?

[ ] Yes, this is a breaking change (please indicate a breaking change in CHANGELOG.md and increment major revision).
[x] No, this is not a breaking change.

This needs to be migrated to apply to the e2e plugin.

Closing this PR as I'm not actively working on it.


xwayland: struts support

https://standards.freedesktop.org/wm-spec/wm-spec-latest.html#idm140200472580736
That would allow X11 bars to work.

_NET_WM_STRUTS_PARTIAL superseeds the normal _NET_WM_STRUTS. But all of this is a bit annoying, and has a few pitfalls.
E.g. bars may (or may not, depends on the client) move around when outputs get moved/resized.
Partial struts on the top end of an output that's below another are really iffy (xmonad/xmobar have a hack for this, that's non-standard).
I think the idea is ok, but I'd rather see work put towards porting applications that would need this to layer-shell, now that it's in rootston/sway (and my experimental waymonad branch).


why f_loss_src operation is trained with f_vars?

In the code the f_loss_src is trained with the f_vars
self.f_train_op_src = slim.learning.create_train_op(self.f_loss_src, self.f_optimizer_src, variables_to_train=f_vars)
but in paper the equation 5 shows that the f_loss_src should trained with g_vars

Did I misunderstand?

@Xztty I remember that i tried to optimize f_train_op_src with the both of g_vars and f_vars but the training process was unstable. Depending on my intuition, I modified the code to resolve the problem.


bench-bot: example commands

bench-bot
Substrate
/cmd queue -c bench-bot $ pallet runtime pallet_name
Example:
/cmd queue -c bench-bot $ pallet dev pallet_contracts
Polkadot
Pallet
/cmd queue -c bench-bot $ pallet runtime pallet_name
Example:
/cmd queue -c bench-bot $ pallet westend-dev pallet_balances
XCM
/cmd queue -c bench-bot $ xcm runtime pallet_name
Example:
/cmd queue -c bench-bot $ xcm westend-dev pallet_balances
Cumulus
Note:  The chain_type argument in the following commands can be one of the folders inside of parachains/runtimes, i.e. as of this writing assets, testing, testing and contracts.
Pallet
/cmd queue -c bench-bot $ pallet runtime chain_type pallet_name
Example:
/cmd queue -c bench-bot $ pallet statemine assets pallet_xcm_benchmarks::generic
XCM
/cmd queue -c bench-bot $ xcm runtime chain_type pallet_name
Example:
/cmd queue -c bench-bot $ xcm statemine assets pallet_xcm_benchmarks::generic

To patch some external branch into the command's branch before running, use the syntax -v PATCH_[repository]=[PR_NUMBER | BRANCH]; for example to patch Substrate PR number 11756 into the command's branch you'd do /cmd queue -v PATCH_substrate=11756 -c bench-bot $ [...args] or to patch branch next into the command's branch you'd do /cmd queue -v PATCH_substrate=next -c bench-bot $ [...args]. This syntax can be specified multiple times for multiple branches to be patched.

Added example for patching feature (https://github.com/paritytech/pipeline-scripts/issues/41)

Updated command syntax for Cumulus (#67)

Docs on pull request commands and queue commend
https://github.com/paritytech/command-bot#pull-request-commands-


MMM Example Notebook returns KeyError: 'dist'

When I tried to run MMM Example Notebook on my Macbook and Colab Enterprise on Vertex AI, I encountered KeyError like below in both environments:

Package versions I used are following.↓
Running on PyMC3 v5.13.1
Running on ArviZ v0.18.0
Running on NumPy v1.26.4
Running on Pandas v2.2.2
Running on PyMC_Marketing v0.5.0

I found that if I don't specify any configuration in my_config, then it works.

Can you able display the model_config attribute for the model instance

sorry, i've read old version example.


Fix link to github prom adaptor, add header links to other packages

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


bug: typescript definitions (index.d.ts) is empty

The index.d.ts included in the distribution of the node module is an empty object {} which means that the constructor cannot be called when writing a webpack config in typescript:
https://webpack.js.org/configuration/configuration-languages/#typescript
This may have to do with the way that the module export is defined.

This will be fixed in 0.6.0


BatchToSpaceND and SpaceToBatchND ERROR_GPU_NOT_COMPATIBLE

1. System information

OS Platform and Distribution (e.g., Linux Ubuntu 20.04):
TensorFlow installation (pip package or built from source): pip
TensorFlow library (version, if pip package or github SHA, if built from source): 2.10.0

2. Code
converter = tf.lite.TFLiteConverter.from_saved_model(in_keras_path)
converter.target_spec.supported_ops = [
    tf.lite.OpsSet.TFLITE_BUILTINS,  # enable TensorFlow Lite ops.
    tf.lite.OpsSet.SELECT_TF_OPS  # enable TensorFlow ops.
]
converter.target_spec.experimental_supported_backends = ["GPU"] # if empty, GPU is not enabled

converter.experimental_new_converter = True

converter.optimizations = [tf.lite.Optimize.DEFAULT] #8-bit quantization
converter.allow_custom_ops = True

tflite_quant_model = converter.convert()

3. Failure after conversion
'BatchToSpaceND' ERROR_GPU_NOT_COMPATIBLE
'SpaceToBatchND' ERROR_GPU_NOT_COMPATIBLE
5. (optional) Any other info / logs
Hi everyone! I am having issues converting the U-2-Net model from PyTorch to TFLite for running on Android. My conversion path is PyTorch(pth) -> ONNX -> TensorFlow -> TFLite. An important requirement is that the TFLite model should support GPU execution and needs to be quantized. The main conversion path works fine, but I encountered an error during the TFLite conversion. After some investigation, I found out that if I change the layers with Conv2D parameters dilation > 1 and padding > 1 to dilation=1 and padding=1, the conversion works without any issues. However, this reduces the model's quality. Obviously, if GPU support is disabled, the model can be converted. I have tried using QuantizationDebugOptions and QuantizationDebugger, but it did not yield any results, the error remains the same. Can you please suggest any way to perform this conversion without compromising the model's quality?

This is my model. You  can do conversion with my code and reproduce my result.
https://drive.google.com/file/d/1nTVfurH8gEjSU8WOx7XHsCqMJwT5g2IT/view?usp=sharing

Hi @ShashmurinSergey, I was able to replicate the issue in Colab using TF v2.10, TF v2.11 and tf-nightly(2.13.0.dev20230305). Please find the gists here(2.10), here(2.11)  and here(tf-nightly). Thank you!

@synandi Yes, replicate is correct, Thank you!
What can you suggest to solve the problem?

Hi @pjpratik!
Thank you for the detailed answer!
Why do these layers appear? After all, everything works correctly when we use dilation=1.
Is there any way for me to convert this model to TFLite with GPU support? Can I somehow remove these layers?

Hi @ShashmurinSergey
They are commonly used in conv2d transpose operation AFAIK.
Have you tried in latest TF 2.11 and TF nightly with dilation!=1 and see if issue still exists?
Thanks.

But I am not using Conv2d transpose, the error occurs on Conv2d layers with dilation=2, 4, 8.
I am using a Docker container with the latest version of TensorFlow, I also tried using a container with tf-nightly image, but I still get the same error.
Is there any other way to convert Conv2d layer with dilation != 1 without adding BatchToSpaceND and SpaceToBatchND layers?

Hi @ShashmurinSergey
Thanks for the information. I have created a toy model with dilation = 1 and dilation = 4. Please find the gist here.
The Conv2D adds the BatchToSpaceND and SpaceToBatchND ops when the dilation rate !=1 for appropriate padding to avoid holes in the output.
Model with dilation = 4

Model with dilation = 1

This can also be observed at
https://github.com/tensorflow/tensorflow/blob/d5b57ca93e506df258271ea00fc29cf98383a374/tensorflow/python/ops/nn_ops.py#L1465
as a result those ops are added which are not compatible for tflite GPU support.
Thanks.

Hi @pjpratik!
Thank for your answer!
Am I correct in understanding that there is no workaround for converting Conv2d with dilation != 1 with GPU support? Could this be possible in the future?

@ShashmurinSergey That could be on roadmap. I am not sure about.
@sachinprasadhs Could you please look into this. Thanks.

Hi @ShashmurinSergey  , if you are able to access a linux system you may be able to resolve your issue by using AI-Edge-Torch, you can find more information here: googleblog.
I have actually created a simple script for converting a model with BathToSpaceND and SpacetoBatchND layers here:
import torch
import torch.nn as nn
import torch.nn.functional as F

class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()

        self.conv1 = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3, stride=1, padding=1)
        self.batch_to_space = nn.BatchNorm2d(16)  # Example batch normalization layer
        self.space_to_batch = nn.ReLU()  # Example activation layer

        self.conv2 = nn.Conv2d(in_channels=16, out_channels=32, kernel_size=3, stride=1, padding=1)

        self.flatten = nn.Flatten()

        self.fc1 = nn.Linear(32 * 32 * 32, 128)  # Adjust input size based on your data shape
        self.fc2 = nn.Linear(128, 10)  # Output layer with 10 classes

    def forward(self, x):

        x = self.conv1(x)
        x = F.relu(x)

        x = self.batch_to_space(x)

        x = self.space_to_batch(x)

        x = self.conv2(x)
        x = F.relu(x)

        x = self.flatten(x)

        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)

        return x

model = MyModel()

input_tensor = torch.randn(1, 3, 32, 32)

edge_model = ai_edge_torch.convert(efficientnet_model.eval(), (input_tensor,))
edge_model.export("simple_conv_.tflite")

If you want to, you can actually try visualizing the result in model-explorer as well.
Please try them out and let us know if this resolves your issue. If you still need further help, feel free to open a new issue at the respective repo.


Is SLIR deprecated in favor of Intervention Request?

On a fork of this project  it says that "SLIR is now deprecated in favor of ambroisemaupate/intervention-request library". Is this true? Should we use that instead?

I haven't used that library before, so I can't comment on if it should be used in favor of SLIR. I don't personally have plans on making updates to SLIR at this time, but if you or someone else wanted to carry the torch I would support that.

Sorry, I don't have the time and my PHP skills are rudimentary. I barely managed #37 by copying existing code... :innocent:

Hi, I choose to stop supporting SLIR because I think it’s not future proof. I found this awesome library http://image.intervention.io/ and I decided to create a wrapper to use it as SLIR, so I created https://github.com/ambroisemaupate/intervention-request. You’ll find the same URL scheme and a caching mechanism but the foundations are built on Composer dependencies and PSR-4 coding standards. It's built to be extensible.


Question: Is the Ballista project providing value to the overall DataFusion project?

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I've noticed from various discussions that Ballista is adding some (considerable?) overhead to the development of DataFusion and it doesn't look like there is much Ballista-specific development happening. Also, the integration tests do not currently appear to be working (https://github.com/apache/arrow-datafusion/issues/1272).
Does it still make sense to continue maintaining Ballista here? I am unable to contribute to Ballista myself at the moment and I don't think the project has reached sufficient momentum for others to be motivated to continue driving it forward, although I haven't been following closely and may have missed some discussions.
The benefits of having Ballista in the codebase are, in my opinion:

It helps keep physical plans serializable
It helps test some of the extension points
It serves as an advanced example for using DataFusion in a distributed context

What do others think?

As an interested user, I would highly appreciate if the datafusion project would keep distributed query execution as a first-class citizen, in the hope that at some point Ballista will replace my PySpark setup.

Keeping distributed computation in mind forces the project to use scalable solutions that make good use of available resources
I can just add more machines to my cluster when I need to run my query on a larger scale

Those two points are the main reason why I turned away from Pandas, Polars, etc. to PySpark.
Even if PySpark is not as fast as e.g. Polars, it does an awesome job on resource management. If I run my notebooks on my laptop, it takes ~20x longer but it will still complete the job.
Compared to pandas which just breaks with OOM when your final dataframe is larger than memory.

TDLR is I think Ballista hold much promise of value, but it needs a champion to push it forward; If it turns into a stagnated / vestigial project it will just be overhead and leaves a great amount of potential value unlocked.

I am unable to contribute to Ballista myself at the moment and I don't think the project has reached sufficient momentum for others to be motivated to continue driving it forward, although I haven't been following closely and may have missed some discussions.

I think this is what Ballista is really missing -- a champion / cheerleader to drive its development forward. It seems to have mostly stalled since being brought into this repo.

The benefits of having Ballista in the codebase are, in my opinion:
It helps keep physical plans serializable

Yes, I agree this is valuable, (and is actually the largest source of overhead in my opinion) -- any change to LogicalPlan or Expr requires being able to update the protobuf serialization / deserialization code.
I think @rdettai  is out for the next few days / weeks but some things he said recently made me think he was planning to focus more on Ballista (perhaps our champion emerges...)
At some point IOx (my company's project) will care about distributed query execution, and at that point we may bring more energy to bear on Ballista, but realistically that is not likely to happen for 6+ more months.

I agree with @alamb that we could really benefit from having someone to actively drive ballista.
Even though I mentioned in my dev list email that ballista introduces extra overhead, I think it acted as a good forcing function for us to design datafusion changes with serailization in mind as you mentioned. For example, It triggered a lot of good discussions around object store serialization in @rdettai 's recent refactoring. If we decided that the overhead is not worth it later, we can just make ballista depend on datafusion by published crate version instead of path so it's less decoupled from datafusion code change. I think this should address all the overhead concerns. I don't think we need to do this right now though.
I think it's best for us to come back to the discuss of moving ballista out of datafusion when it gets significant momentum.

Thanks @andygrove for starting this discussion. I did feel some overhead induced by DataFusion on some of my recent PRs, especially from the serialization part (but not only). I was really happy to do that work, because I find it very interesting to see the internals of a distributed engine, but I could understand that it might be seen as an unpleasant constraint for contributors with limited bandwidth and no interest in the distributed system.
@alamb I am considering trying to merge Ballista with Buzz, but that represents a huge amount of work, because Ballista would require a fair amount of restructuring/modularization for it to work. I would be happy to take the challenge and push Ballista forward, but I would require some sort of sponsorship to work on it in a sustained fashion 😅. I have some leads already, but if you have any idea who might be willing to invest into this I am highly interested.

Thanks for all the feedback so far. I think @alamb makes a very good point about Ballista needing a champion to drive it forward and I agree with that. Let's see what happens there.

I am coming to this discussion from a different perspective.  I recently wrote a py-sparkling](https://github.com/svenkreiss/pysparkling)-inspired RDD that distributes list slices over MPI (https://github.com/frobnitzem/mpi_list).
However, loading up pandas dataframes quickly wastes memory.  So, I'm investigating arrow as a memory-friendly replacement.  The trouble I am running into is that DataFusion might have too much functionality.  My csv files are already split up (many per process), and I already have processes running on an existing cluster via MPI.  So I want to execute SQL queries once for each csv file and create a new result dataset distributed the same way as the original.

I would be happy to take the challenge and push Ballista forward, but I would require some sort of sponsorship to work on it in a sustained fashion 😅. I have some leads already, but if you have any idea who might be willing to invest into this I am highly interested.

I don't have anything for you at the moment @rdettai  but I'll keep my ears open. @andygrove do you have any ideas (specifically any users / potential users of Ballista that might be willing to contribute time)?

The trouble I am running into is that DataFusion might have too much functionality. My csv files are already split up (many per process), and I already have processes running on an existing cluster via MPI. So I want to execute SQL queries once for each csv file and create a new result dataset distributed the same way as the original.

@frobnitzem I think this is a bit off topic and worth to be discussed in a separate issue. To answer your question, you can just use datafusion as a simple library to query a single csv file in process using sql. You don't have to use ballista.

I recently started doing some work around DataFusion and seeing good potential with Ballista to become a candidate of our next-gen analytics engine. At this point I am trying to put in some missing pieces and get it working with some benchmark cases we are using for initial validation. If all goes well the plan is to start a team to help drive this forward. I am really interested in working with anyone who share similar interest to kick start the discussion. Please let me know if there are interests and we can use this thread the form an initial group. @andygrove it would be really helpful if we can pick your brain from time to time, and if you have time maybe share some of the things you were thinking to add.
My personal opinion is that Ballista makes sense to be part of DataFusion. It may potentially affect (and depend on) many things such as partition strategy, join strategy, planner, etc. I recognize there might be some overhead in development, maybe this is an opportunity for us to define some APIs to make it easier. @alamb @houqp Do you guys think this is reasonable?
@rdettai I am hoping you are still interested in this topic. Do you think the challenges you had can be solved by introducing an API layer between DataFusion and Ballista and we make some effort to keep it somewhat stable?
I am working on a high level proposal right now and hopefully can share in the next few weeks. Please feel free to let me know if you have any ideas, I am currently working on this besides my full time job so may need a little time.

This is great news @realno ! Thank you for driving this. I am interested in getting ballista to work with delta tables managed in S3 and potentially use it to replace some of the spark SQL jobs we have at work as a starting point. However, I am also just working on datafusion and ballista in my spare time, so the output is limited.
As for reducing development overhead, ballista currently only depends on public datafusion APIs, so if you have good ideas on how to restrict the api coupling between ballista and datafusion, then it's certainly a good thing. I haven't seen anymore express strong opinion on the overhead yet, so I don't think we are at a stage where we need to invest heavily on this. It is something we can keep an eye on while iterating on ballista.

This is great news @realno ! Thank you for driving this. I am interested in getting ballista to work with delta tables managed in S3 and potentially use it to replace some of the spark SQL jobs we have at work as a starting point. However, I am also just working on datafusion and ballista in my spare time, so the output is limited.
As for reducing development overhead, ballista currently only depends on public datafusion APIs, so if you have good ideas on how to restrict the api coupling between ballista and datafusion, then it's certainly a good thing. I haven't seen anymore express strong opinion on the overhead yet, so I don't think we are at a stage where we need to invest heavily on this. It is something we can keep an eye on while iterating on ballista.

Thanks @houqp for the insight. This is good to hear there are already public APIs in place. And my hope is very aligned with yours that to replace some Spark Jobs at some point :) I will reach out once I have some thing ready for review/discussion, feel free to let me know if anything I can help with too.

I recently started doing some work around DataFusion and seeing good potential with Ballista to become a candidate of our next-gen analytics engine. At this point I am trying to put in some missing pieces and get it working with some benchmark cases we are using for initial validation. If all goes well the plan is to start a team to help drive this forward. I am really interested in working with anyone who share similar interest to kick start the discussion. Please let me know if there are interests and we can use this thread the form an initial group. @andygrove it would be really helpful if we can pick your brain from time to time, and if you have time maybe share some of the things you were thinking to add.
My personal opinion is that Ballista makes sense to be part of DataFusion. It may potentially affect (and depend on) many things such as partition strategy, join strategy, planner, etc. I recognize there might be some overhead in development, maybe this is an opportunity for us to define some APIs to make it easier. @alamb @houqp Do you guys think this is reasonable?
@rdettai I am hoping you are still interested in this topic. Do you think the challenges you had can be solved by introducing an API layer between DataFusion and Ballista and we make some effort to keep it somewhat stable?
I am working on a high level proposal right now and hopefully can share in the next few weeks. Please feel free to let me know if you have any ideas, I am currently working on this besides my full time job so may need a little time.

look forward to your proposal.
If you need help or discussion, please feel free to let me know.

@realno @liukun4515 do you have any plans for ballista that you think are worth adding to the Q1 roadmap?

@realno @liukun4515 do you have any plans for ballista that you think are worth adding to the Q1 roadmap?

@matthewmturner
Sorry for my limited time, Maybe I can be involved in Q2 or later.
In the Q1, I have other things to do, but I will try my best to track this.

My personal opinion is that Ballista makes sense to be part of DataFusion. It may potentially affect (and depend on) many things such as partition strategy, join strategy, planner, etc. I recognize there might be some overhead in development, maybe this is an opportunity for us to define some APIs to make it easier. @alamb @houqp Do you guys think this is reasonable?

I think this is very reasonable. I view datafusion as a single node, sharable query engine that isn't really designed to be used by itself -- rather it can be used to create systems such as ballista (and other projects)
Having ballista in the same repo as a way to validate datafusion API changes seems valuable to me 👍  and I am happy to have some minor extra overhead on DataFusion API changes. What I really want to see is Ballista actively used / driven forward by the community, which it sounds like @realno  is preparing to propose.
cc @yahoNanJing and @gaojun2048 who have opened PRs with contributions for Ballista recently

@alamb Actually I'm quite curious on the point of datafusion not being used standalone.
On my side, my plan was to use datafusion (likely via the Python bindings) until my data size warranted using ballista. I thought it was a nice selling point that I could use the same underlying engine for either single node or distributed compute.
Is having first class support for using datafusion standalone at odds with it being a component of a larger system?
Also aren't the datafusion Python bindings meant to enable using it standalone?

@realno @liukun4515 do you have any plans for ballista that you think are worth adding to the Q1 roadmap?

@matthewmturner It is a bit ambitious to answer this question at this moment. My plan is to have a high level proposal first, and I am pretty sure it will be going through a couple of iterations based on community feedback. I think then would be a good time to talk about roadmap and timeline.

@realno ok! IMHO, I think that even mentioning that a plan is being developed is worth mentioning just so we can show that there are plans for ballista development even if they are just in the planning stage. The alternative makes it look like Ballista isn't getting any focus when that isn't the case.
So if okay with you I can put something like "Putting together design docs, plan, and priorities for Ballista"

@alamb Got it - thanks for clarifying!

Late to the party here but my team is very excited about the potential of Ballista and are interested in helping push the project forward.

Given the recent development in ballista, I would say the answer to this question is not an unequivocal "YES" and this ticket doesn't need to remain open


Use ruff in vscode

Can you test if it works for you?
For me, after some struggle, it works well. A nice improvement is that it will sort the imports.
https://github.com/huggingface/datasets-server/assets/1676121/ba698454-b6df-42d6-b402-676b6d030e8c

fixes #2044


Fix deprecated compiler options warning

Hi,
I forked your project after finding your issue here for adding test coverage. When I opened the solution in VS I noticed a compiler warning in project.json - this pull request fixes that issue.
I will be looking to write some tests in the next week or so, in the mean time I hope this PR gets merged.
Nick

Yep, that's probably an artifact of the changes the .NET team made between RC1 and RC2.
Thanks!
-Steve


Script mode set -xe by default when no shebang used

Changes
Fixes #1735
When I use script mode without a #! the current behaviour is to insert
one for me - #!/bin/sh. This is helpful and removes an unneccessary
line from my script. In addition I'd like two things by default: First
I want the script to error out if any of the commands error out. Second
I'd like to see the commands the script runs before their output. These
two features are baked into sh with the set -xe flags.
After this change the default behaviour of script mode when no shebang
is provided is to insert both the default shebang as well as the flags
required to make shell errors and shell echos happen.

Submitter Checklist
These are the criteria that every PR should meet, please check them off as you
review them:

[x] Updates tests (if functionality changed/added)
[x] Commit messages follow commit message best practices

Reviewer Notes
If API changes are included, additive changes must be approved by at least two OWNERS and backwards incompatible changes must be approved by more than 50% of the OWNERS, and they must first be added in a backwards compatible way.
Release Notes
Script mode will now insert `set -xe` to scripts that don't include a shebang #!, which will in turn fail scripts with commands that error our and also print the command that is about to run. This default behaviour can be disabled by simply specifying your own #! at the top of the script.

Can you document this behavior at https://github.com/tektoncd/pipeline/blob/master/docs/tasks.md#step-script too?

Can you document this behavior at https://github.com/tektoncd/pipeline/blob/master/docs/tasks.md#step-script too?

Good call, done!

I1211 16:24:52.372] ----------------------------------------------
I1211 16:24:52.372] ---- Checking links in the markdown files ----
I1211 16:24:52.373] ----------------------------------------------
I1211 16:24:54.221] docs/tasks.md
I1211 16:24:54.221] 	ERROR	https://en.wikipedia.org/wiki/Shebang_(Unix
I1211 16:24:54.221] 		Not Found (HTTP error 404)

Hrm.  Second time I've seen this today.  The website works when I access it from a browser.
/test pull-tekton-pipeline-build-tests

I1211 16:24:52.372] ----------------------------------------------
I1211 16:24:52.372] ---- Checking links in the markdown files ----
I1211 16:24:52.373] ----------------------------------------------
I1211 16:24:54.221] docs/tasks.md
I1211 16:24:54.221] 	ERROR	https://en.wikipedia.org/wiki/Shebang_(Unix
I1211 16:24:54.221] 		Not Found (HTTP error 404)

Hrm. Second time I've seen this today. The website works when I access it from a browser.
/test pull-tekton-pipeline-build-tests

The link check might be not as smart as we are hoping for :stuck_out_tongue:

The link check might be not as smart as we are hoping for

Yeah, its picking up the URL as : https://en.wikipedia.org/wiki/Shebang_(Unix which does indeed return a 404 response

/test pull-tekton-pipeline-build-tests

/test pull-tekton-pipeline-build-tests