Should ExecForEach sanitize or escape arguments differently?

After a recent conversation about script on Lobste.rs, I'm concerned about how ExecForEach handles arguments (especially filenames?) with tricky characters (e.g., spaces, single quotes, and double quotes).
The current example for ExecForEach is ListFiles("*").ExecForEach("touch {{.}}").Wait(). I'll take that as a baseline, and start with the following minimal script:
package main

import "github.com/bitfield/script"

func main() {
	script.ListFiles("*").ExecForEach("touch {{.}}").Stdout()
}

Imagine that lives in a folder with the following files:
$ ls -l
total 24
-rw-r--r--  1 telemachus  wheel     0 Aug 21 07:14 Bobby" Tables"
-rw-r--r--  1 telemachus  wheel     0 Aug 21 07:14 Bobby' Tables'
-rw-r--r--  1 telemachus  wheel   210 Aug 21 07:17 go.mod
-rw-r--r--  1 telemachus  wheel  3221 Aug 21 07:17 go.sum
-rw-r--r--  1 telemachus  wheel     0 Aug 21 07:19 hello world
-rw-r--r--  1 telemachus  wheel   125 Aug 21 07:16 main.go

Here's the result of running the script:
$ ls -l
total 24
-rw-r--r--  1 telemachus  wheel     0 Aug 21 09:32 Bobby Tables
-rw-r--r--  1 telemachus  wheel     0 Aug 21 07:14 Bobby" Tables"
-rw-r--r--  1 telemachus  wheel     0 Aug 21 07:14 Bobby' Tables'
-rw-r--r--  1 telemachus  wheel   210 Aug 21 09:32 go.mod
-rw-r--r--  1 telemachus  wheel  3221 Aug 21 09:32 go.sum
-rw-r--r--  1 telemachus  wheel     0 Aug 21 09:32 hello
-rw-r--r--  1 telemachus  wheel     0 Aug 21 07:19 hello world
-rw-r--r--  1 telemachus  wheel   124 Aug 21 09:32 main.go
-rw-r--r--  1 telemachus  wheel     0 Aug 21 09:32 world

I would expect (hope?) that script would sanitize the filenames, so that touch ran once with each filename (including the ones with tricky characters).
Instead, here are some things that go wrong (I think).

touch seems to receive the argument Bobby Tables twice. In a way, this is a good result: a single name with a space! On the other hand, all quotes seem to be stripped away from the two original filenames, one of which has single quotes and one of which has double quotes. If you change the code to script.ListFiles("*").ExecForEach("ls {{.}}").Stdout(), you'll see ls: Bobby Tables: No such file or directory (without quotes of any kind) twice in the output. These results suggest that something script does (the templating, perhaps?) removes quotations from filenames.
ExecForEach seems to split the filename hello world into two arguments so that the program runs touch hello and touch world.

Normally, in actual shell scripting, you could fix problems like these with proper quoting.
for file in *
do
	touch "$file"
done

But I can't see any way to do the equivalent using script. Ideally, script itself would handle this for the user. (E.g., to protect users who don't know about such dangers from themselves.)
I noodled around with this a bit last night using https://bitbucket.org/creachadair/shell and https://github.com/mvdan/sh, but nothing I did helped. I'd be interested in working on this, but at the moment I can't see a way forward.

Thanks for putting so much thought into this, @telemachus—it's most helpful.
I think I understand what's going on here now. Let's take the spaces issue first. Something like this:
script.Echo("my file").ExecForEach("touch {{.}}").Wait()

doesn't work as expected, because we end up with two files: my and file. Well, this wouldn't work in the shell either, as we've discussed, so the solution would be to quote the filename part of the command like this:
script.Echo("my file").ExecForEach("touch \"{{.}}\"").Wait()

That takes care of the spaces issue, but what about files with quotes in their names?
script.Echo("my \"quoted\" file").ExecForEach("touch \"{{.}}\"").Wait()

This creates a file named my quoted file, which isn't quite right: it should be literally my "quoted" file. The Go template parser is eating the quotes. Again, the solution is to escape the quotes before they get interpolated:
script.Echo("my \\\"quoted\\\" file").ExecForEach("touch \"{{.}}\"").Wait()

But when the list of filenames comes from some arbitrary source, what can we do? This must be a problem whenever filenames are interpolated into a Go template, mustn't it?
Honestly, I'll confess this to you, but keep it to yourself: I don't much care for Go templates. I try not to use them if at all possible. However, if we can figure out the exact right way to rewrite arbitrary data in script pipes such that it expands to the correct values in templates, I'd certainly consider adding it to the package.
So, to coin a phrase, PRs are welcome!

This problem has the same shape as the classic problem of HTML escaping: how do you make sure 1 < 2 turns into exactly 1 < 2 and not 1 &lt; 2 etc. The thorough solution would be to do what the Go html/template engine does: have a separate type for "safe" strings and turn raw strings into safe strings before final output. That seems like too much effort though. A hacky but easy solution is to add a function to the template that looks like this:
type SafeString string

func quote(s any) SafeString {
  switch v := s.(type) {
  case SafeString:
     return v
  case string:
    return SafeString(shell.Quote(v)
  default:
   panic("bad type!")
 }
}

Then you use it on the client side like .ExecForEach(`touch {{.|quote}}`.

You mean this one ?
https://github.com/google/safehtml

It sounds like we have a plan!

It sounds like we have a plan!

Sorry to be dense, but what is it? Best guess: is the plan for someone to write up a PR with (something like?) Carl's quote helper? If so, should the docs recommend that people always use that when they feed arguments to external commands? (I think that means that people should always use the helper when they use ExecForEach, but I am not 100% sure about that.)

is the plan for someone to write up a PR with (something like?) Carl's quote helper?

That certainly seems like it would be a good value-add, doesn't it? You've constructed the failing test case, @carlmjohnson has pointed out what the solution should look like, and @gedw99 has identified that the necessary code already exists. Between us, I think we've got this!

If so, should the docs recommend that people always use that when they feed arguments to external commands? (I think that means that people should always use the helper when they use ExecForEach, but I am not 100% sure about that.)

Yes, at first I thought we should simply always quote-sanitise input data to the template, but that's not quite good enough, because we don't know that the data contains filenames, for example. Therefore, it needs to be up to the user to call this function in their template if they're interpolating filenames.
Adding this to the documentation should also appease the wrath of @matklad:

What I do find objectionable is not mentioning this in the docs for a function!

Some gopherology from by an ex-crustacean:
It feels like there should be a way to configure default escaping rule for Template, because that's what you'll just need for more or less any use-case. And, like, there is html escaping, it should work somehow?
And, look, indeed, Templae exports the underlying parse tree:
https://pkg.go.dev/text/template#Template
type Template struct {
	*parse.Tree
	// contains filtered or unexported fields
}

So I guess that's the way! You could walk the parse tree manually, applying your custom escaping rules.
Now, how do I actually use that?

Template is the representation of a parsed template. The *parse.Tree field is exported only for use by html/template and should be treated as unexported by all other clients.

Oh...
:-)

In the current version of Go, if you put a package inside a directory called internal, it becomes unimportable by external packages. The Go team has said the parse tree for templates would have been in internal if internal had existed when it was created, but they don't want to move it now. Cf. https://github.com/golang/go/issues/25357

@winks, I agree that Go template syntax isn't easy to love. But my thinking is that it's better to use something standard, perfect or not, than to invent my own way of doing this.

Second thing, yes find ... -print0 and xargs -0 is kinda clunky, but see above, if you're wrapping this up in the lib, wouldn't that be a possibility to implicitly use something like that?

Can you elaborate on this point a little bit? I'm not sure I understand what you're referring to.


Add ability to ask additional questions for event (interview) signup

It would be helpful to be able to add additional questions on a per-event basis.
E.g. For interviews, we would like to be able to ask if the interviewer(s) will be able to bring a tablet / laptop. (To be able to view the electee's survey response and progress)

explain why including the directive to do so (bring a laptop, etc.) in the description is insufficient

Reading something in a description and actually having to check a button,
make a selection from a pull down menu, or something similar are quite
different. Things in the description can be missed. Having a question that
is required will force the user to provide the information one way or the
other.
As far as for interviews, having a question of "Will you be able to bring a
tablet or laptop" would be helpful as we attempt to pair interviewers with
and without. So yes, it really is better to have the option to add
additional event signup questions on a per event or per event type basis.
Michael L. Benson
Ph.D. Candidate, Radiation Laboratory
The University of Michigan
Department of Electrical Engineering & Computer Science
1301 Beal Av. #3214
Ann Arbor, MI 48109-2122
Phone:<PHONE_NUMBER>
Email<EMAIL_ADDRESS>On Wed, Jan 14, 2015 at 12:44 PM, mikehand<EMAIL_ADDRESS>wrote:

explain why including the directive to do so (bring a laptop, etc.) in the
description is insufficient
—
Reply to this email directly or view it on GitHub
https://github.com/tbpmig/mig-website/issues/161#issuecomment-69957991.

You're not pairing people though. People are just signing up for shifts...


[ADAP-565] Seed direct dependencies of delta models (from state diff)

Is this your first time submitting a feature request?

[X] I have read the expectations for open source contributors
[X] I have searched the existing issues, and I could not find an existing issue for this feature
[X] I am requesting a straightforward extension of existing dbt-bigquery functionality, rather than a Big Idea better suited to a discussion

Describe the feature
My boss asked me to investigate if this workflow is possible:
delta_nodes = Slim CI changed nodes

sseds = []
for dbt_model in delta_nodes:
    model_deps = get_direct_deps(dbt_model) # how??

for seed in seeds:
   create_seed_in_test_env(seed)

dbt -x run —select delta_nodes

i’m curious whether this is a solved problem.
Describe alternatives you've considered
Run thw whole project on every change
Who will this benefit?
Cli users who use slim ci
Are you interested in contributing this feature?
I suspect this one is too hard for me
Anything else?
No response

Thinking about it more, I expect that you're gonna tell me to use a paid subscription to dbt cloud to access the Metadata API. That seems like it's in your best interests. Is that right?

I'd empower you with knowledge about the Metadata API if that were the best way to solve the problem, but I'm not sure if that's the case here or not. I wasn't fully understanding some key details in your original, so that's why I asked for clarification.
For feature requests like this, we just try to understand the problem you're trying to solve and then determine if it's in the scope of dbt-core or not. Sometimes the request is already possible in dbt-core. Other times it's out of scope. And plenty often we accept the idea and give guidance on how we'd want to see it implemented.

I found the deferrals feature.
It looks very similar to what I'm looking for.


Raspberry Pi

Very very impressed with this iteration of smart mirror software. I currently have a Raspberry Pi 2 as the main board for my smart mirror. Is there any way to port this code to the Raspberry Pi? Can you create a version for the Raspberry Pi? Thanks in advance

It might work if you run android things,  https://developer.android.com/things/hardware/raspberrypi.html


ES-1946 Update CODE FREEZE approvers list

Adding Owen for P2P code freeze approvers

Can you add David Currie also?

Can you add David Currie also?

I'll raise a separate PR for this


Error messages: generate as many as we can, make them consistent

All of our error messages should be consistent in terms of capitalisation, spelling, person, tense, etc. At the moment there is some degreee of variation.
We should come up with a standard. The sooner we do this the fewer incorrect messages we'll have to change, but it's really not a high priority.

Error messages in general will always need improvement, but things have improved a lot since this issue was opened. I don't think there's a need for this issue now, so closing.


Move regex to new file function and general refactor

Created new valid_word_check.py
Refactored main.py
Amended charDict in word_to_score.py to use * for the blank tiles

refactoring a little more - will submit new PR


[Azure cognitive Search] Fix example of get group ids to remove non sql cosmos heads

Latest improvements:
MSFT employees can try out our new experience at OpenAPI Hub  - one location for using our validation tools and finding your workflow.

Contribution checklist:

[ ] I have reviewed the documentation for the workflow.
[ ] Validation tools were run on swagger spec(s) and have all been fixed in this PR.
[ ] The OpenAPI Hub was used for checking validation status and next steps.

ARM API Review Checklist

[ ] Service team MUST add the "WaitForARMFeedback" label if the management plane API changes fall into one of the below categories.
adding/removing APIs.
adding/removing properties.
adding/removing API-version.
adding a new service in Azure.

Failure to comply may result in delays for manifest application. Note this does not apply to data plane APIs.

[ ] If you are blocked on ARM review and want to get the PR merged urgently, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.
Please follow the link to find more details on API review process.

/azp run automation - sdk

Can one of the admins verify this patch?

#sign-off

LGTM, Is the PR ready to merge?

@ruowan  - Yes. thank you!

LGTM, Is the PR ready to merge?

Yes please, could you merge it?


failed to "KillPodSandbox"delegateDel: error invoking ConflistDel - "k8s-pod-network": conflistDel: error converting the raw bytes into a conflist: error parsing configuration list: no 'plugins' key"

What happend:
I Specify default cluster network in Pod annotations following guide here  https://github.com/intel/multus-cni/blob/master/doc/configuration.md#specify-default-cluster-network-in-pod-annotations
When I delete pod, there are errors like following:
error killing pod: failed to "KillPodSandbox" for "xxxxx" with KillPodSandboxError: "rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"xxxxxx\" network: delegateDel: error invoking ConflistDel - \"k8s-pod-network\": conflistDel: error converting the raw bytes into a conflist: error parsing configuration list: no 'plugins' key"

What you expected to happen:
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
Environment:

Multus version
3.4.1
image path and image ID (from 'docker images')
Kubernetes version (use kubectl version):
1.17.9
Primary CNI for Kubernetes cluster:
calico
OS (e.g. from /etc/os-release):
File of '/etc/cni/net.d/'
File of '/etc/cni/multus/net.d'
NetworkAttachment info (use kubectl get net-attach-def -o yaml)
Target pod yaml info (with annotation, use kubectl get pod <podname> -o yaml)
Other log outputs (if you use multus logging)

Can you provide the NetworkAttachmentDefinition that you used? And if possible enable logging and include the logs? Thanks!
Logging info here @ https://github.com/intel/multus-cni/blob/master/doc/configuration.md#logging
Also, if you don't mind include the contents of /etc/cni/net.d/00-multus.conf.
I looks like some configuration, somewhere, isn't parsing correctly.
Thanks!

I have the same problem. If i specify sriov is default network, error appear when delete pod. If i use flannel as default network, error dont appear.

this is because that delegates configured in 00-multus.conf but v1.multus-cni.io/default-network is used in pod annotation.
		// check plugins field and enable ConfListPlugin if there is
		for _, v := range in.Delegates {
			if len(v.ConfList.Plugins) != 0 {
				v.ConfListPlugin = true
			}
		}

two workarounds:

use "clusterNetwork" instead of delegates in 00-multus.conf, something like
{  "cniVersion": "0.3.1",  "kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig",  "logFile": "/var/log/multus.log",  "logLevel": "debug",  "name": "multus-cni-network",  "type": "multus",  "confDir": "/etc/cni/net.d",  "multusNamespace": "secure-container",  "clusterNetwork": "flannel" }

update your net-attach-def CR to use plugins, something like
apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: kuryr-test namespace: secure-container spec: config: '{ "cniVersion": "0.3.1", "name": "kuryr", "debug":true, "plugins":[ {"type": "kuryr-cni"} ] }'

this is because that delegates configured in 00-multus.conf but v1.multus-cni.io/default-network is used in pod annotation.
		// check plugins field and enable ConfListPlugin if there is
		for _, v := range in.Delegates {
			if len(v.ConfList.Plugins) != 0 {
				v.ConfListPlugin = true
			}
		}

two workarounds:

use "clusterNetwork" instead of delegates in 00-multus.conf, something like
{  "cniVersion": "0.3.1",  "kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig",  "logFile": "/var/log/multus.log",  "logLevel": "debug",  "name": "multus-cni-network",  "type": "multus",  "confDir": "/etc/cni/net.d",  "multusNamespace": "secure-container",  "clusterNetwork": "flannel" }

update your net-attach-def CR to use plugins, something like
apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: kuryr-test namespace: secure-container spec: config: '{ "cniVersion": "0.3.1", "name": "kuryr", "debug":true, "plugins":[ {"type": "kuryr-cni"} ] }'


Issue 319 todos

Simplifying filter method:

Moved highlight function into HighlightKeyword class

#319 related

Current coverage is 84.83% (diff: 78.57%)

Merging #367 into master will decrease coverage by 0.02%

@@             master       #367   diff @@
==========================================
  Files            35         35
  Lines          4314       4319     +5
  Methods           0          0
  Messages          0          0
  Branches        394        395     +1
==========================================
+ Hits           3661       3664     +3
- Misses          611        616     +5
+ Partials         42         39     -3

Powered by Codecov. Last update ec36d4c...7ef31a4

👍


Process author avatar via gatsby-image

Since this type isn't a MediaItem, it has no local file.
This could easily be pulled in to be processed via Sharp
{
  wpPage {
    author {
      avatar {
        url
      }
    }
  }
}

👀

hi @TylerBarnes and future gatsby-source-wp devs, i found a solution
i forked my solution following these gatsby instructions for processing remote images  from the using-gatsby-source-wordpress-experimental repo:
https://github.com/TylerBarnes/using-gatsby-source-wordpress-experimental/compare/master...shawngrona:avatar-image
above works for me but curious - when you opened this pr were planning to include this functionality automatically in the source wp plugin?

We recently moved this plugin over to the gatsbyjs monorepo and are trying to clean out this issue queue. We documented this feature to look at for a future roadmap item and are going to close this issue out. If you want to add additional visibility to this feature, you can request it at https://portal.gatsbyjs.com/


Closing file/folder open dialogs

I believe there should be a feature to close the open file/folder dialogs. It's annoying whenever I accidentally go to "open folder" when I'm simply trying to open a file and I end up having to open the working folder AGAIN and wait for code-server to load just to get rid of the dialog.

You can press ESC to close the dialog.


typing.ByteString and compatibility with Sequence[int]

Environment data

Language Server version: Pylance language server 2022.3.1 (pyright bd47ac0b) starting
OS and version: Ubuntu 20.04.4 LTS
Python version: 3.8

Expected behaviour
typing.ByteString should work as a type for variables that should hold some kind of bytes-like object such as bytes, bytearray or memoryview and only require read-only access
If not this type then I would expect some reasonable way of typing such objects while working with them without forcing a specific type.
Normal operations on the variables such as concatenation and slicing should preserve this without requiring casts.
Actual behaviour
Can't add any of the listed types to a typing.ByteString
foo = typing.cast(typing.ByteString, b'foo')
foo + b'bar'

"""
Operator "+" not supported for types "ByteString" and "Literal[b'bar']"
  Operator "+" not supported for types "ByteString" and "Literal[b'bar']"
"""

Slicing a ByteString returns typing.Sequence[int] which is not compatible with ByteString
bar = typing.cast(typing.ByteString, b'bar')[:]

"""
Expression of type "Sequence[int]" cannot be assigned to declared type "ByteString"
  "Sequence[int]" is incompatible with "ByteString"
"""

This makes it very unergonomic to work with bytebuffers while keeping correct typing as any operation must explicitly cast the type back to ByteString to stay type-compatible.

I do understand there is some problem in that there really isn't any way in python to represent a sequence of ints limited to 0-255
but since this case is very common when working with any binary protocol having a type that's specialized for it feels motivated and is what I would have expected ByteString to be.
If nothing else I file this to have the problems documented since I was surprised I couldn't find any existing bugs or discussion of it.

Ok checking some in the mypy issues where it also has come up it seems like typing.ByteString is not really complete as it should and the general recommendation is to just use bytes for annotations as allowed by https://docs.python.org/3/library/typing.html#typing.ByteString
I see that pylance generally accepts this between bytes and bytearray but memoryview objects are still marked as incompatible.
I personaly find it clearer and would prefer an Interface type such as ByteString to document that variants are allowed but if memoryviews would be accepted as a bytes subclass that would at least allow a consistent typing.

The type definition for ByteString comes from the typing.pyi stub in the typeshed project. Pyright (the type checker that underlies Pylance) bundles the typeshed stubs for stdlib type information. If you right click on ByteStream and choose "Show Declaration", Pylance will take you to the relevant declaration in the stub file.
All of the major Python type checkers, including pyright and mypy, use the same typeshed type stubs. If you think that there is an error in these stubs, please file a bug or a PR in the typeshed project. Pyright (and Pylance) regularly pick up changes to this repo.


Issue/improve instrumentation for race

Description
Attempt to improve error reporting on race condition
Added potential fix

Processing this pull request

Pull request rejected by merge tool. The tests for this branch did not succeed.

Processing this pull request

Pull request rejected by merge tool. The tests for this branch did not succeed.

Processing this pull request

Merged into branches master in 8c6fb0d8346af5f6ad0b9f48dc37cbe378453961


fix typo

Merging this PR will publish the following release:

2.2.1 (2019-10-29)
Bug Fixes

add link to npmjs and release rollup peerep (c85ec68)

Merging this PR will publish the following release:

2.2.1 (2019-10-29)
Bug Fixes

add link to npmjs and release rollup peerep (c85ec68)

Merging this PR will publish the following release:

2.2.1 (2019-12-28)
Bug Fixes

add link to npmjs and release rollup peerep (c85ec68)

Merging this PR will publish the following release:

2.2.1 (2019-12-28)
Bug Fixes

add link to npmjs and release rollup peerep (c85ec68)

Thanks @epicfaace!

:tada: This PR is included in version 2.2.3 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Update style-and-voice.md

Small grammar error "contain a more details" changed to "contain more details"

This seems like a small (but important) contribution, so no Contribution License Agreement is required at this point. We will now review your pull request.
Thanks,
Microsoft Pull Request Bot

@jkoseattle : Thanks for your contribution to the Azure documentation! The author, @azdocprs, has been notified to review your proposed change.

@jkoseattle : Thanks for your contribution to the Azure documentation! The author, @azdocprs, has been notified to review your proposed change.

@tysonn Hi Tyson Please review this proposed update to the contributor guide. Thanks.


Notification of rule changes

We all have a bunch of abaplint.json files in our projects and want to keep them up-to-date with the latest abaplint rules. It would be nice to have some kind of notification or method to identify which rules are new and which have changed.
Maybe add a "last update" date to the metadata of each rule and surface it on rules.abaplint.org so we can filter the rules by it.

hi,
the best solution right now is to open abaplint.json in vscode, it will help entering new rules and add new configuration options, according to the json schema

but did something break, I'm trying to add more stuff without directly breaking existing
however might do changes, where eg. rules are renamed, and thus not run on old configurations

It was a couple renames that threw me off. No worry

currently, I'm prioritizing functionality over stability for abaplint


memory related updates

add helpers select_bank and memcpy_bank1
move memory layout specifications all into MCLMemory.h; update memory layout.

question: all the track data goes to bank1 (and shared parts in the internal memory), what's in bank0?

Let me explain the memory.
The ATMEGA2560 has 8KB of SRAM and is capable of addressing 64KB of memory at a time.
With the external SRAM connected  you get 8KB (internal) + 56KB (external) (8KB is always wasted per bank)
avr-gcc is used to specify the memory layout upon linking.
The internal SRAM 8KB has been reserved for the program heap. This means, memory (variables + objects) defined in function calls will be limited to 8KB. Good thing about the heap is that the memory is instantaneously reclaimed when you exit a function.
The remaining 56KB is bank selectable and is used by Global variables and Objects.
You could theoretically use this space for calls to malloc() but this is a bad idea on a small microprocessor, due to RAM fragmentation. It is better to layout all the memory you are going to use ahead of time, and dont change it.
Bank Switching:
The MC features 2 banks. Bank 0 and Bank 1. Bank 0 is the default bank is used to store data that is essential to program function.
Some examples of data stored in Bank 0: The MIDI and SYSEX buffers. The sequencer data. These data structures are accessed in ISR()s and should be as quick to access as possible.
Bank 1, is mainly used for cached data. For example, in chain mode, tracks are transferred from the SD card to bank 1, for faster load times.
Because many of the MIDICTRL data structures are stored Globally, switching banks and then trying to access these structures will cause the MC to crash. You need to be more clever. When switching banks, only memory in the heap is reliable.
Data is thus syphoned in and out of second memory bank to an object in the heap, like so:
379 bool MDTrackLight::load_from_mem(uint8_t column, uint32_t region) {
380   uint32_t len = sizeof(GridTrack) + sizeof(MDSeqTrackData) + sizeof(MDMachine);
381
382   uint32_t pos = region + len * (uint32_t)column;
383
384   volatile uint8_t *ptr;
385
386   ptr = reinterpret_cast<uint8_t *>(pos);
387   // cbi(TIMSK0, TOIE0);
388   switch_ram_bank(1);
389   memcpy(this, ptr, len);
390   switch_ram_bank(0);
391   // sbi(TIMSK0, TOIE0);
392   return true;
393 }

The second memory bank (bank 1) memory is laid out manually. I use this space or cached elements.
Anything that needs to be accessed quickly I will aim to keep in (bank 0)

Revised my memory explanation for a second time. I mistakenly used the word heap in place of stack, which would be rather confusing.

Thanks for explaining about the memory layout! Yeah I get the idea that starting from 0x2200 are selectable banks, I’ve seen this before in sram test (that’s why marking the pointers volatile is a good idea, similar to PROGMEM segment).
wrt the select_bank macro, it establishes a guard instance RamBankSelector, which restores old bank during destruction. This way it ensures that bank is restored when the current stack frame is popped(and the local instance goes out of life cycle) — this is pretty common for modern c++ (e.g. std::lock_guard) as RAII pattern. The benefit is that, for example, during ISR, you could return midway without worrying about manual bank restoration— the destructor does this.
Anyway, I do think this is a trade off here, because it hides bank selection details, and sometimes it may not be the best idea.

Another question: I see the difference of MDTrack vs. MDTrackLight;
Only MDTrackLight is placed in bank1, the remainder, the PLocks, stay in the “heap” (yeah I get this) — is that part a shared data structure by 16 tracks?

Regarding MDTrack and MDTrackLight.
"is that part a shared data structure by 16 tracks?"
In Chain mode. MDTrack objects are read from the SDCard and cached in to bank2.
Chain mode does not use the MD's sequencer information.
MDTrackLight Class was used to create a smaller object with only the MC's internal sequencer data.
The larger MDTrack data structure is used for Write/Save functionality. An MDTrack object includes all the data for a MD track including some Kit settings. This data structure is unique per track, and none of the  memory elements are shared.

"wrt the select_bank macro, it establishes a guard instance RamBankSelector, which restores old bank during destruction. This way it ensures that bank is restored when the current stack frame is popped(and the local instance goes out of life cycle) — this is pretty common for modern c++ (e.g. std::lock_guard) as RAII pattern. The benefit is that, for example, during ISR, you could return midway without worrying about manual bank restoration— the destructor does this."
Cool. As you would have noticed, I haven't employed destructors. But this seems like an excellent application and might provide some additional flexibility to ISR()s in the future.
I will review the rest of your changes later tonight/tomorrow.
Thanks for the contribution. If anything, the code needs to be aligned with best practices.
My time is limited (and sometimes i much rather be making music than dev), so help here is much appreciated.

Ok that checks out. The extra data is not cached in the high bank.
If my memory layout specs hold, then it’s possible to pack in a few more tracks right, I’d like to get A4 FX/CV tracks for controlling a MS-20 mini.

Expanding to 6 x A4 Tracks should be do-able.
You need to decide if the CV + FX settings are important, and should they be stored alongside the sequencer data. ( I think they should ).
I haven't looked at the A4 sysex in a while, If i recall, CV + FX settings will be different from the sound spec.
Right now we don't do any actual decoding of the A4 data. We just store and send the sound dumps as they are. If you want to build a proper A4 pattern + kit + sound decoder then be my guest :-)
I know there may be some stack memory issues with write_mode as defined in MCLActions.
See the poorly named "MCLActions::send_pattern_kit_to_md()"
Basically, the A4 sounds are cached in the stack, and then sent to the A4 all in one hit.
We may need to move this stack memory to bank2, to fit 6 tracks in.

On a more broader topic. I had always planned for tracks higher than 16, to be general MIDI sequencer tracks. The functionality is 90% there. Though the sequencer pages need to be modified to support entering note data in using the encoders.

You need to decide if the CV + FX settings are important, and should they be stored alongside the sequencer data. ( I think they should ).

I think they should. And my guess is that they share the same data structure with normal tracks -- substituting Osc1 with CV-A, Osc2 with CV-B, Filters with CV-C and Amp with CV-D.. Let me dig around here.
The CV machines have different options (other than the parameters, options are accessed with Fn+page) though -- CV control type (trig/gate etc.), polarity, voltage (up to 10v!). Gotta be careful 0.0

btw I've rebased this PR upon dev and I'm done with the memory-related changes.
I created the Bank1Object class so that the bank1 accessors can be further unified.

In MidiUart.cpp:
#include <MidiClock.h>
MidiUartClass MidiUart;
MidiUartClass2 MidiUart2;

In main.cpp:
uint8_t sysexBuf[5500];
MidiClass Midi(&MidiUart, sysexBuf, sizeof(sysexBuf));
uint8_t sysexBuf2[2800];
MidiClass Midi2(&MidiUart2, sysexBuf2, sizeof(sysexBuf2));

In platform.txt:
compiler.cpp.flags=-c -g -Os {compiler.warning_flags} -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -MMD -flto -Wl,--gc-section,--defsym=__stack=0x8020ff,--section-start,.data=0x82100,--defsym=__heap_end=0x80ffff -I{build.core.path}/Midi -I{build.core.path}/GUI -I{build.core.path}/CommonTools -I{build.core.path}/Elektron -I{build.core.path}/MD -I{build.core.path}/MidiTools -I{build.core.path}/Sequencer -I{build.core.path}/MNM -I{build.core.path}/PageTools -I{build.core.path}/MDSketches -I{build.core.path}/MDPages -I{build.core.path}/A4 -I{build.core.path}/SPI -I{build.core.path}/Wire -I{build.core.path}/Adafruit-GFX-Library -I{build.core.path}/Adafruit_SSD1305_Library -I{build.core.path}/SdFat -I{build.core.path}/SDCard -I{build.core.path}/MCL

Ok I get this. We're telling the compiler "it's okay, relax, and put things beyond 0x2200" -- super cool!

In the interest of time, i've got ahead and merge your commits to dev branch.
I've reverted the changes to GridRowHeader.* (this is a bug and needs to be fixed next time the project spec changes in master)
The changes to bank switching (using object destructor) need to be tested.

In main.cpp we use  ISR_NOBLOCK for timer2. This prevents missed reads of the UART buffers, by re-enabling global interrupts inside the ISR.
If ISR(TIMER2_COMPA_vect) is interrupted after select_bank(0), will the destructor   ~RamBankSelector() will be called?

It won’t be called on interruption, because an interruption is effectively the same as a “thread being swapped out in the OS”, and does not change the current instruction pointer(when later coming back), and thus not changing the life scope of locals.
The destructor will be triggered after restoration, when the (first) ISR exits.

Excellent. Thanks.

Another thing I slightly worry about the memory layout is the alignment issue.
In a beefier machine where the words are wider, the compiler would leave paddings between members of a class, and thus making the size larger than the sum of the members.
Not sure what’s the case with avr-gcc and the 8bit mcu

Reference: https://www.avrfreaks.net/forum/packed-struct
The potential problems are mitigated by adding attribute((packed)) before the class keyword.

Right. My assumption was that avr was not padding. For the classes i've checked manually in the past, sizeof was expected.
But I could be wrong here.

Nope, I worry more about overlapped tracks. That’ll be a problem.
Yatao

From: jmamma<EMAIL_ADDRESS>Sent: Monday, March 25, 2019 12:15 PM
To: jmamma/MIDICtrl20_MegaCommand
Cc: Yatao Li; Author
Subject: Re: [jmamma/MIDICtrl20_MegaCommand] memory related updates (#2)
Right. My assumption was that avr was not padding. For the classes i've checked manually in the past, sizeof was expected.
But I could be wrong here.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjmamma%2FMIDICtrl20_MegaCommand%2Fpull%2F2%23issuecomment-476049182&data=02|01|yatli%40microsoft.com|a749d1628bb64a7396d008d6b0d878e3|72f988bf86f141af91ab2d7cd011db47|1|0|636890841117479930&sdata=Rcf8TsR24gifJ7j4LU4N71fkNpphGfTv5G52hroJ%2FzU%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FATufsKRsUSHxy4CwAdkhMGlg7nzSSIwzks5vaE1NgaJpZM4cFB7l&data=02|01|yatli%40microsoft.com|a749d1628bb64a7396d008d6b0d878e3|72f988bf86f141af91ab2d7cd011db47|1|0|636890841117489943&sdata=3y%2BhwGxwk7cmui1izW6%2FBRycJ9NWlQxyUqkzpknx8nA%3D&reserved=0.

My understanding was that sizeof() should return correct object size, including any potential padding.
Given that we use len = sizeof() this should mitigate collisions ?

I can take care of this. sizeof gives faithful results, but I’d need to adjust Bank1Object a bit.
Yatao

From: jmamma<EMAIL_ADDRESS>Sent: Monday, March 25, 2019 12:31 PM
To: jmamma/MIDICtrl20_MegaCommand
Cc: Yatao Li; Author
Subject: Re: [jmamma/MIDICtrl20_MegaCommand] memory related updates (#2)
My understanding was that sizeof() should return correct object size, including any potential padding.
Given that we use len = sizeof() this should mitigate collisions ?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjmamma%2FMIDICtrl20_MegaCommand%2Fpull%2F2%23issuecomment-476051458&data=02|01|yatli%40microsoft.com|76dc8133d1234507dfe708d6b0daceee|72f988bf86f141af91ab2d7cd011db47|1|0|636890851153807883&sdata=EsxBUPT12GVUAF0vNQjslh%2BlvEVGbj%2BnDpdwe8EpfRQ%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FATufsIi1cJySTEYxUK6VjjGr2DbkkgTaks5vaFE5gaJpZM4cFB7l&data=02|01|yatli%40microsoft.com|76dc8133d1234507dfe708d6b0daceee|72f988bf86f141af91ab2d7cd011db47|1|0|636890851153817888&sdata=mgze6D6GFPh6U%2FAB25bLASFsAal19MdyssuPKb43HgI%3D&reserved=0.

Bank1Obj is updated. This mixin is now able to access the true size of the attached class.


swipeToSlide jumps back to first slide

Hi there, first thanks for a great slider.
I am currently experiencing an issue with the swipeToSlide function.
Not sure if anyone else has experienced this but occasionally when I drag the slider from left to right it jumps back to the first slide.
If you check out my codepen below and click/drag from number 1 and drag off the screen to the right it jumps back, however it doesn't do this in the opposite direction
http://codepen.io/AshWood/pen/ojeJQj?editors=100

Downgraded to version 1.4.1 as issue isn't present. It seems to be present for me in 1.5.0 and above

I found that if you had swiped right to left first and then do left to right, then it doesn't jump back.

@st11x Did you manage to find a solution to the problem? I've been messing about with the values and settings but so far haven't been able to solve it. It does only seem to affect swiping from the first slide to the right by more than one slide.

@AshleyPothecary. No I did not.I wasn't sure about the side effects of changing those lines back. Also, version 2.0 has the same newer lines.
Reverting to version 1.4.1 was worse for me as there is a flash of (slide-to-show -  slides-to-scrolll) slides each time I swipe.  I am lazy loading the images so it could be only when that is enabled.

@st11x Seems to work fine if the slider has multiple rows... I'll wait and see if the issue is resolved in the next update

@kenwheeler Issue still seems present in 1.5.9, has anyone else reported a similar issue to this or is it just myself that has this issue?

Hey. The issue is bugging me as well. It shows exactly like you described, AshleyPothecary.
Happens only when the carousel is in intial state. Even if it returns to that state after a full iteration. Always reproducible.
Looking forward to a fix. At this point in time I am not familiar enough with the code to try a fix by myself.

Looking into this

Still happening with the master.

I am on the 1.6 version but I realize that there's a cheap fix to this if you comment out or remove the
//max = _.slideCount * 2;
although you have to play around with the options to make sure that you are showing enough slides a time or when it hit the 2nd endpoint the whole thing breaks. I am not sure if that's what you are looking for, if you encounter any problem keep posting here.


feat: bump salt config version

Context

Relates with https://github.com/odpf/salt/pull/66
To fix telemetry headers configuration via environment variable

should we wait until the salt PR is merged into main and then bump to that version?

latest salt version are breaking changes https://github.com/odpf/salt/pull/63, it requires to make changes, let's do that on separate PR, wdyt @rahmatrhd ?


[SPARK-34590][TESTS] Allow JDWP debug for tests

What changes were proposed in this pull request?

This PR proposes a new feature that allows developers to debug test code using JDWP with sbt an Maven.
More specifically, this PR introduces the following profile options.

jdwp-test-debug: An profile which controls enable/disable JDWP debug
test.jdwp.address: An option which corresponds to address option in JDWP
test.jdwp.suspend: An option which corresponds to suspend option in JDWP
test.jdwp.server: An option which corresponds to server option in JDWP
test.debug.suite: An option which controls whether debug ScalaStyle suites (Maven only)

For sbt, this feature can be used like build/sbt -Pjdwp-test-debug -Dtest.jdwp.address=localhost:9876 -Dtest.jdwp.suspend=y -Dtest.jdwp.server=y and can be used for both JUnit tests and ScalaTest tests.
For Maven, this feature can be used like as follows:
(For JUnit tests) build/mvn -Pjdwp-test-debug -Dtest.jdwp.address=localhost:9876 -Dtest.jdwp.suspend=y -Dtest.jdwp.server=y
(For ScalaTest suites) build/mvn -Pjdwp-test-debug -Dtest.debug.suite=true -Dtest.jdwp.address=localhost:9876 -Dtest.jdwp.suspend=y -Dtest.jdwp.server=y (It might be useful to specify specific sub-modules like -pl sql/core,sql/catalyst).
Why are the changes needed?

It's useful to debug test code.
Does this PR introduce any user-facing change?

No.
How was this patch tested?

I confirmed the following things.

jdwp-tes-debug can switch JDWP enabled/disabled
test.jdwp.address can change address and port.
test.jdwp.suspend can change the behavior that the target debugee suspends or not.
test.jdwp.server can change the behavior that the JDWP debugger run as a server or client.
ScalaTest suites can be debugged with Maven with setting test.debug.suite to true.

Test build #135638 has started for PR 31706 at commit a609cd9.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40218/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40218/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40218/

Test build #135638 has finished for PR 31706 at commit a609cd9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135638/

@dongjoon-hyun It's just I don't know how to do the same thing for Maven for now. I'll add the same feature after I find the way.

It would just be a new profile that modifies some test runner settings or something

@srowen
Thanks for the advice. Now we can switch this feature with -Pjdwp-test-debug both for sbt and Maven.
@dongjoon-hyun
For Maven, scalatest-maven-plugin supports JDWP debug but it's a little bit unuseful (suspended for each sub-modules even though a sub-module doesn't contain suites to run) so I've added test.debug.suite property only for Maven.

I think it's OK to just add for SBT.

Test build #135683 has started for PR 31706 at commit d0461ee.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40265/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40265/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40265/

Test build #135683 has finished for PR 31706 at commit d0461ee.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135683/

I think it's OK to just add for SBT.

I usually use SBT so it's OK for me but @dongjoon-hyun , do you think it's OK even if this change doesn't contain this feature for Maven?

Merged to master. Thank you, @sarutak and @srowen .


(RFC) WIP: Use Go Generate instead of ORM

This is a potential alternative to #136 (or it could be used in conjunction with it) and builds upon the work that @amorenoz did for adding bindings to libovsdb.
An ORM API is mainly useful when you are storing abstract data in database. You would create your structs in Go, and the ORM API would deal with creating tables, mapping go types to database types and vice versa etc...
However, as are dealing with data that is described in a Schema having ORM is, in my opinion, less useful.
Our structs in Go, need to mirror that schema exactly.
Therefore, I propose that instead of using an ORM API, we simply us Golang's code generation to generate the bindings.
This has got a couple of advantages:

It relies on a very minor change to libovsdb to make a method public
It would be easy to extend the tool to generate multiple packages (e.g v4, v5` etc..) for handling differing schema versions should we want to minimise the impact of API changes for library consumers
It removes human error, by generating all tables/fields of the schema automatically

Here is an example of what is generated:
/* THIS CODE WAS AUTOGENERATED BY GENFRONSCHEMA */
/* DO NOT EDIT */

package ovn_northbound

import (
	"fmt"
	"reflect"
)

const (
	ACLTableName = "ACL"
)

type ACL struct {
	UUID        string
	Match       string
	Severity    []string
	Meter       []string
	Name        []string
	Direction   string
	Action      string
	Log         bool
	ExternalIds map[string]string
	Priority    int
}

func (odbi *ovndb) rowToACL(uuid string) (*ACL, error) {
	s := &ACL{
		UUID: uuid,
	}
	cache, ok := odbi.cache[ACLTableName][uuid]
	if !ok {
		return s, fmt.Errorf("object does not exist in cache")
	}
	sPtr := reflect.ValueOf(s)
	if sPtr.Type().Kind() != reflect.Ptr {
		return s, fmt.Errorf("not a struct ptr")
	}
	obj := reflect.Indirect(sPtr)

	if v, ok := cache.Fields["match"]; ok {
		err := odbi.colToField("match", v, obj.FieldByName("Match"))
		if err != nil {
			return s, err
		}
	}

	if v, ok := cache.Fields["severity"]; ok {
		err := odbi.colToField("severity", v, obj.FieldByName("Severity"))
		if err != nil {
			return s, err
		}
	}

	if v, ok := cache.Fields["meter"]; ok {
		err := odbi.colToField("meter", v, obj.FieldByName("Meter"))
		if err != nil {
			return s, err
		}
	}

	if v, ok := cache.Fields["name"]; ok {
		err := odbi.colToField("name", v, obj.FieldByName("Name"))
		if err != nil {
			return s, err
		}
	}

	if v, ok := cache.Fields["direction"]; ok {
		err := odbi.colToField("direction", v, obj.FieldByName("Direction"))
		if err != nil {
			return s, err
		}
	}

	if v, ok := cache.Fields["action"]; ok {
		err := odbi.colToField("action", v, obj.FieldByName("Action"))
		if err != nil {
			return s, err
		}
	}

	if v, ok := cache.Fields["log"]; ok {
		err := odbi.colToField("log", v, obj.FieldByName("Log"))
		if err != nil {
			return s, err
		}
	}

	if v, ok := cache.Fields["external_ids"]; ok {
		err := odbi.colToField("external_ids", v, obj.FieldByName("ExternalIds"))
		if err != nil {
			return s, err
		}
	}

	if v, ok := cache.Fields["priority"]; ok {
		err := odbi.colToField("priority", v, obj.FieldByName("Priority"))
		if err != nil {
			return s, err
		}
	}

	return s, nil
}

// TODO: Implement Add/Delete functions and the inverse of rowToX

NOTE: I've currently set this to generate in to a ovn_northbound package.
Once i've implemented the remaining functions, I plan to generate these files in the repository root, adjust client.go appropriately and hopefully all the tests would still pass.
Should #136 be preferred by the maintainers, it would be relatively easy to generate all of the structs automatically using a small program like this.

Hi @dave-tucker, thanks for this nice idea.
I thought of doing this as well. Although I think the ORM approach has some useful properties, I agree that some of them might not be urgently needed.
My main concerns are two:

Try to avoid API explosion, : try to avoid having LSList, LRGet, .... and instead have a generic List(). Same goes for the callback API (OnLogicalRouterCreated()). The current limitation makes the library unusable for many monitoring purposes.

I achieve this by creating a Model interface that all types implement and that can be used to properly cast the struct on the callback function as in the example.
Similarly, generic List(), Get(), Create()... functions can be added using only one parameter (typically, the pointer to the object to be filled).
This can, of course, be done using your approach (e.g: adding the table name to the functions and using interface{} as the return values). Just putting it out there for discussion and feedback :)

Enable the generated code to live outside go-ovn.

My fear was (is) that if go-ovn ties itself to a particular version of ovn (schema), we might be creating new problems for users that want to use the library for other versions. We could decide to only track stable ovn branches but backporting fixes to the ovn-tracking branches would increase maintenance cost. By having the generated code outside go-ovn, it's the client (e.g: ovn-kubernetes) the one who decides what version of go-ovn and what version of ovn (i.e: what schema) to use. I wrote a small ORM model generator in https://github.com/amorenoz/ovnmodel to test my experiments that demonstrates this (the generator code itself needs a complete cleanup but it demonstrates the point).
What do you think?

Also interested in @palanik1's take

Thanks @dave-tucker ! Code generation for ovsdb schema is what we have always wanted to do, so I definitely agree with the idea, and I think with @amorenoz 's work in libovsdb we are very close. In fact I was thinking the next step after https://github.com/eBay/libovsdb/pull/31 is to have a code generation tool in libovsdb to generate API implementations for a given schema based on the ORM API.
So first of all I think this tool should be in libovsdb. In fact I am not quite sure if there is still a need for go-ovn to exist as a separate repo after that - maybe just a wrapper on top of / managing the generated APIs. I'd like to hear your inputs.
Secondly, I think code generation is not conflict with ORM API. Instead, it can be based on the ORM API. What do you think? @dave-tucker @amorenoz

Try to avoid API explosion, : try to avoid having LSList, LRGet, .... and instead have a generic List(). Same goes for the callback API (OnLogicalRouterCreated()). The current limitation makes the library unusable for many monitoring purposes.

Ack. This won't avoid exploding the API. In fact it will have the opposite effect as we'll automatically have bindings for all the tables 😢
Another thing that needs to be solved is

My fear was (is) that if go-ovn ties itself to a particular version of ovn (schema), we might be creating new problems for users that want to use the library for other versions....

In my experience, you're damned if you do, damned if you don't 😉
If we push the responsibility to users, we'll need the ability to constrain the schema version in the config... otherwise we run the risk of a user hitting an incompatible OVN schema version at runtime and the library won't work as expected. There may be varying ovn versions in the wild depending on which release train of which linux distro you're running on.
On the other hand, we don't have to have loads of branches here... I think (although I haven't totally thought this through) that a library user could choose the major version (i.e import github.com/eBay/go-ovn/ovn_northbound/v5/`) and we could (optionally) get fancy and support fallback if the OVN version doesn't meet the required schema level.
Disclaimer: This is theoretical

So first of all I think this tool should be in libovsdb. In fact I am not quite sure if there is still a need for go-ovn to exist as a separate repo after that - maybe just a wrapper on top of / managing the generated APIs. I'd like to hear your inputs.

I think the generation tool could live in libovsdb. Then go-ovn would become little more than the generated structs and some glue code... unless (see comment above) there is a desire to get fancy and handle things like mismatched client versions.

Secondly, I think code generation is not conflict with ORM API. Instead, it can be based on the ORM API.

Agreed. You can use it with, or without. I really don't have a strong opinion about it.
I'll go ahead and leave some comments on #145 though

Closing this PR for two reasons.

This functionality should live in libovsdb
Having spent a little more time today playing with this... it's clear the API offered by #136 is much, much better


Build fails with "/bin/sh: ./build.sh: not found"

Hello,
I have tried to follow the README on Docker Desktop for Windows, but the build failed:
PS C:\dev\GIT> docker build --build-arg LOGSPOUT_VERSION=master -t mycustomlogspout .
unable to prepare context: unable to evaluate symlinks in Dockerfile path: CreateFile C:\dev\GIT\Dockerfile: The system cannot find the file specified.
PS C:\dev\GIT> cd .\logspout-fluentd
PS C:\dev\GIT\logspout-fluentd> docker build --build-arg LOGSPOUT_VERSION=master -t mycustomlogspout .
Sending build context to Docker daemon  180.1MB
Step 1/17 : ARG LOGSPOUT_VERSION=master
Step 2/17 : FROM gliderlabs/logspout:${LOGSPOUT_VERSION} as logspout
master: Pulling from gliderlabs/logspout
aad63a933944: Already exists                                                                                64743c68eac1: Pull complete                                                                                814834b814fe: Pull complete                                                                                Digest: sha256:9a9387443cd48cd6f0cbf4af3c829a81f9a9692af316cd67534ac5afc3d0b740
Status: Downloaded newer image for gliderlabs/logspout:master
Executing 3 build triggers
---> Running in 305e38eefb73
/bin/sh: ./build.sh: not found
The command '/bin/sh -c cd /src && chmod +x ./build.sh && sleep 1 && sync && ./build.sh "$(cat VERSION)-custom"' returned a non-zero code: 127

Had the same issue on windows. My solution was to change the fileendings from crlf to lf of the build.sh and it worked.

In my case, I just removed the build.sh file. then re-create it, so it's work.


Create REST blueprint not returning full expected response.

I noticed I was not getting the same response on a post as I was on a get. Specifically, associations.
Referencing 'Create new record while associating w/ another new record'
https://github.com/balderdashy/sails-docs/blob/master/reference/blueprint-api/Create.md
I recreate the scenario from the documentation with sails new testProj, then sails generate api pony and sails generate api pet, then update the models to have a one-to-one association.
Pony
module.exports = {
    attributes: {
        pet: {
            model: 'pet'
        }
    }
};

Pet
module.exports = {
    attributes: {
        pony: {
            model: 'pony'
        }
    }
};

Then, sails lift away.
POST /pony
JSON Req
{
  "name": "Pinkie Pie",
  "hobby": "ice skating",
  "pet": {
    "name": "Gummy",
    "species": "crocodile"
  }
}

JSON Res
{
  "name": "Pinkie Pie",
  "hobby": "ice skating",
  "pet": 1,
  "createdAt": "2014-11-21T19:18:09.161Z",
  "updatedAt": "2014-11-21T19:18:09.161Z",
  "id": 1
}

As you can see, the response did not include the full pet object, just its id. In the documentation, it shows the full object being returned.
The entries and the association were definitely created successfully,
GET - /pony/1
{
  "pet": {
    "name": "Gummy",
    "species": "crocodile",
    "createdAt": "2014-11-21T19:18:09.157Z",
    "updatedAt": "2014-11-21T19:18:09.157Z",
    "id": 1
  },
  "name": "Pinkie Pie",
  "hobby": "ice skating",
  "createdAt": "2014-11-21T19:18:09.161Z",
  "updatedAt": "2014-11-21T19:18:09.161Z",
  "id": 1
}

Is this an error in documentation, or an error in code? Either way, is there a way we can specify that we want the associations to be returned?
One-to-one associations are just showing up as the linked id, many to many do not show up at all. I was looking into the toObject() function for waterline for another issue when I saw the showJoins option; it looks like it might be related to that.

I noticed that too
https://github.com/balderdashy/sails-docs/issues/384

Is this still an issue?


Unable to fetch committed metadata when I restart the job

When I redeploy the spark streaming job, I notice this issue where the job fails to fetch committed metadata from Dynamodb. I set the startingposition to TRIM_HORIZON. The job is fine if I delete all the checkpoint entries in the dynamo table and restart it.
Any idea what might be causing this issue.
24/08/12 19:42:14 WARN ProcessingTimeExecutor: Current batch is falling behind. The trigger interval is 30000 milliseconds, but spent 176311 milliseconds
24/08/12 19:42:14 INFO KinesisV2MicrobatchStream: get latestOffset prevBatchId 2
24/08/12 19:42:14 INFO DynamodbMetadataCommitter: exists check from table stream_processor for batchId 2
24/08/12 19:42:14 INFO DynamodbMetadataCommitter: apply query limit 1.
24/08/12 19:42:14 INFO DynamodbMetadataCommitter: queryDataByBatchId- started from table stream_processor for batchId 2 with limit Some(1).
24/08/12 19:42:14 INFO DynamodbMetadataCommitter: queryDataByBatchId- lastEvaluatedKey was null - scan finished for batchId 2.
24/08/12 19:42:14 INFO DynamodbMetadataCommitter: queryDataByBatchId- Listed 0 item(s) from table stream_processor for batchId 2 with limit Some(1).
24/08/12 19:42:14 INFO package: reportTimeTaken queryDataByBatchId took 6 ms
24/08/12 19:42:14 INFO KinesisV2MicrobatchStream: getBatchShardsInfo for batchId 2 doesn't exists
24/08/12 19:42:14 ERROR MicroBatchExecution: Query [id = ....., runId = ......] terminated with error
java.lang.IllegalStateException: Unable to fetch committed metadata from previous batch id 2. Some data may have been missed
	at org.apache.spark.sql.connector.kinesis.KinesisV2MicrobatchStream.latestOffset(KinesisV2MicrobatchStream.scala:231)
	at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$constructNextBatch$4(MicroBatchExecution.scala:491)
	at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:427)
	at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:425)
	at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:67)
	at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$constructNextBatch$2(MicroBatchExecution.scala:490)
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
	at scala.collection.Iterator.foreach(Iterator.scala:943)
	at scala.collection.Iterator.foreach$(Iterator.scala:943)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
	at scala.collection.IterableLike.foreach(IterableLike.scala:74)
	at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
	at scala.collection.TraversableLike.map(TraversableLike.scala:286)
	at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
	at scala.collection.AbstractTraversable.map(Traversable.scala:108)

It shows batch 2 data is not in dynamodb.  When you use dynamodb to save kinesis connector's metadata,  Spark Structured Streaming's checkpoint data (committed, offset, etc.) is still in the checkpoint location specified by your job.  You need to ensure the checkpoint data and dynamodb metadata are consistent.

ah, that makes sense. thanks for the quick clarification.


Fix idle timeout dialogue appearance since Material upgrade

Added MatDialogTitle, MatDialogContent, MatDialogActions  as imports to app.module.


Kaggle TPU: TFLite-Model-Maker installation issue

Issue type
Bug
Have you reproduced the bug with TensorFlow Nightly?
Yes
Source
source
TensorFlow version
2.9.3
Custom code
No
OS platform and distribution
Kaggle TPU
Mobile device
No response
Python version
3.8.16
Bazel version
No response
GCC/compiler version
No response
CUDA/cuDNN version
No response
GPU model and memory
TPU VM v3-8
Current behavior?

Installed TFlite-model-maker using command !pip install -q tflite-model-maker
Tried to print the installed version using command import tflite_model_maker print(tflite_model_maker.__version__)
Got error ImportError: libusb-1.0.so.0: cannot open shared object file: No such file or directory
I am attached here the error log.
kaggle_tpu_error.txt

Standalone code to reproduce the issue
In Kaggle, enable TPU accelerator.
Install the TFLite Model Maker library.
Print the version of installed library.

Relevant log output
No response

Hi @dsbyprateekg, tflite-model-maker installation is currently broken and it's unlikely to be resolved soon, in the mean time please try using mediapipe-model-maker instead: here is an example gist. Let us know if for some reason you can't use mediapipe model maker to accomplish your goals. For now this is considered a duplicate of https://github.com/tensorflow/tensorflow/issues/60431

Hi @dsbyprateekg, does the above work around work for you?

@pkgoogle I think the broken installation issue is still there in platforms like colab and kaggle. Since it has been many months, any timeline when it will be fixed?

Hi @dsbyprateekg, since that doesn't work, please try using MediaPipe-ModelMaker instead


style: use fqcn for task plugins names

Another PR to use fqcn in project and make ansible-lint complain without it

Thanks for this !