mavlink send helpers refactor to prevent writing partial messages

Possible quick work around for https://github.com/PX4/Firmware/issues/12957.
This pull request rearranges the mavlink helpers slightly to shift the the buffer, should_transmit, error count updates into the single final mavlink mavlink_end_uart_send call rather than each mavlink_send_uart_bytes (called 3 times per mavlink message send).
This is accomplished by unifying some of the uart/udp logic differences. Each write (mavlink_send_uart_bytes) goes into a buffer, then that buffer is written out once (mavlink_end_uart_send). This should prevent broken mavlink messages from going out over the wire, and will also be slightly more efficient by minimizing work for each send.
TODO:

[ ] move the get_free_tx_buf() check to mavlink_start_uart_send, store the result and skip mavlink_send_uart_bytes if there's not enough buffer space left.

TODO: review MavlinkStream::get_size()
Does this need to be updated for mavlink v1, v2, v2 + signing?

For context this is the imposed mavlink library helper structure we're dancing around.
https://github.com/mavlink/c_library_v2/blob/ac40c0329e88b70ae5db4c1467ed5853d305af54/mavlink_helpers.h#L352-L359
	MAVLINK_START_UART_SEND(chan, header_len + 3 + (uint16_t)length + (uint16_t)signature_len);
	_mavlink_send_uart(chan, (const char *)buf, header_len+1);
	_mavlink_send_uart(chan, packet, length);
	_mavlink_send_uart(chan, (const char *)ck, 2);
	if (signature_len != 0) {
		_mavlink_send_uart(chan, (const char *)signature, signature_len);
	}
	MAVLINK_END_UART_SEND(chan, header_len + 3 + (uint16_t)length + (uint16_t)signature_len);

Can you also please copy the PR description in to the commit? (Or squash it so it goes into the git history.)

One of the reasons I like "Squash and merge" is we get one last chance to clean it up.

Tested on CUAC+ V5:
Modes Tested
Position Mode: Good.
Altitude Mode: Good.
Stabilized Mode: Good.
Mission Plan Mode (Automated): Good.
RTL: Good.
- Procedure
Arm and Take off in position mode, after flying for approximately one minute, switched to altitude then stabilized mode proceed to switch to mission plan mode then make sure that vehicle follows all waypoints as shown in QGC, once completed all waypoin activate RTL and see landing behaviour.
Notes:
No issues noted, good flight in general.
Log:
https://review.px4.io/plot_app?log=25cc3296-2428-41dc-a6a4-04e1c6251f7c

tested on  pixhawk4 v5 f-450
on mission mode
altitude mode
no issues
log:
https://review.px4.io/plot_app?log=77ed5ecb-c1fc-4b66-be47-4007dc83640d

@dagar we tried to test on PixRacer V4, no connection by the telemetry on QGC. We tried stable and it worked fine.

Looks like I screwed up the rebase when extracting the other PRs. Attempting to fix.

The only way I can think of getting around the extra memory is for each message, while still using the helper functions, is to allocate a buffer bigger than just the payload data, use the mavlink_msg_XXXXX_out_dynamic_send_buf helper instead, with an offset into the buffer, and then let it copy the data back onto itself. :nauseated_face:

and will also be slightly more efficient by minimizing work for each send.

It is not more efficient since you have another memcpy.

Yes, but overall it's still slightly faster, although not enough to get excited about.
master (top)
PID COMMAND                   CPU(ms) CPU(%)  USED/STACK PRIO(BASE) STATE FD
167 mavlink_if0                  3900  6.137  1824/ 2532 100 (100)  w:sig  4
168 mavlink_rcv_if0               255  0.327  2424/ 3916 175 (175)  w:sem  4

PR (top)
PID COMMAND                   CPU(ms) CPU(%)  USED/STACK PRIO(BASE) STATE FD
167 mavlink_if0                  2153  5.400  1816/ 2532 100 (100)  w:sig  4
168 mavlink_rcv_if0               178  0.409  2608/ 3916 175 (175)  w:sem  4

master (mavlink status)
instance #0:
        GCS heartbeat:  394441 us ago
        mavlink chan: #0
        type:           USB CDC
        flow control: OFF
        rates:
          tx: 23.249 kB/s
          txerr: 0.000 kB/s
          tx rate mult: 1.000
          tx rate max: 800000 B/s
          rx: 0.527 kB/s
        FTP enabled: YES, TX enabled: YES
        mode: Config
        MAVLink version: 2
        transport protocol: serial (/dev/ttyACM0 @2000000)
        ping statistics:
          last: 0.81 ms
          mean: 0.92 ms
          max: 103.86 ms
          min: 0.27 ms
          dropped packets: 0

PR (mavlink status)

instance #0:
        GCS heartbeat:  8932 us ago
        mavlink chan: #0
        type:           USB CDC
        flow control: OFF
        rates:
          tx: 23.367 kB/s
          txerr: 0.000 kB/s
          tx rate mult: 1.000
          tx rate max: 800000 B/s
          rx: 0.527 kB/s
        FTP enabled: YES, TX enabled: YES
        mode: Config
        MAVLink version: 2
        transport protocol: serial (/dev/ttyACM0 @2000000)
        ping statistics:
          last: 0.51 ms
          mean: 0.94 ms
          max: 27.88 ms
          min: 0.23 ms
          dropped packets: 0

It also requires considerably more memory (I see almost 1KB on pixracer) due to the additional buffer.
Can we please find a better solution? I know the mavlink API is not great for that.

Unless you have another idea I don't think the memory is that much of a concern. This buffer is 280 bytes (MAVLINK_MAX_PACKET_LEN) per mavlink instance. I'd still much rather fix this in mavlink, and I even considered pulling more of that code PX4 side. We can come back to this later if the mavlink api situation changes/improves.
This doesn't really excuse the waste, but we're still going to be somewhere around 5-10 K ahead of v1.9.2 on baseline configurations, and the cost of each additional module is lower. If we consider the system overall and the annoying constraints like the mavlink api I think I'd rather invest the time in other areas of the system where we can easily save quite a bit more than 1 KB (finish uORB updates, further reduce FDs, drop NuttX HPWORK/LPWORK, move more tasks to WQs, minimize logger subscriptions array, etc).

Tested on PixRacer V4:
Modes Tested
Position Mode: Good.
Altitude Mode: Good.
Stabilized Mode: Good.
Mission Plan Mode (Automated): Good.
RTL: Good.
- Procedure
Arm and Take off in position mode, after flying for approximately one minute, switched to altitude then stabilized mode proceed to switch to mission plan mode then make sure that vehicle follows all waypoints as shown in QGC, once completed all waypoin activate RTL and see landing behaviour.
Notes:
No issues noted, good flight in general.
Log:
https://review.px4.io/plot_app?log=768fc7c2-7bab-4a92-8161-24a44a2ca8bc
https://review.px4.io/plot_app?log=dabe9570-73cf-4f91-886b-e9be0cc4f58b

Yes, but overall it's still slightly faster, although not enough to get excited about.

I saw different numbers then.

Unless you have another idea

Given the existing mavlink API we can move the get_free_tx_buf() check to mavlink_start_uart_send, store the result and skip mavlink_send_uart_bytes if there's not enough buffer space left. This avoids partial writes, but still requires 3 writes, which I see as the less important issue.

Given the existing mavlink API we can move the get_free_tx_buf() check to mavlink_start_uart_send, store the result and skip mavlink_send_uart_bytes if there's not enough buffer space left. This avoids partial writes, but still requires 3 writes, which I see as the less important issue.

Good idea.

Updated to check the buffer initially and skip copying, but not yet tested.

How will this affect log downloads (especially over FTDI/non-usb links) and log streaming?

This is still using the additional buffer and copy. Did you forget to push?

This came up during the dev call, what's the current state is it ready to go in? @dagar

It turns out that this PR can prevent PARAM_VALUE messages from getting dropped when the ground station requests them.
I tested this using just FTDI without flow control connected to Telem1 using a small tester based on MAVSDK: https://github.com/mavlink/MAVSDK/pull/1038
For the results, see: https://docs.google.com/spreadsheets/d/1aAT58x1zVd6_5bL9NjCOnc87Eoy4Aa3QMd-cmwZXI8A/edit?usp=sharing

@julianoes very cool test you did there.
Just out of curiosity, does anyone know why the loading time doesn't decrease further moving from 115200 to 460800? Is there some kind of rate limiter?

Just out of curiosity, does anyone know why the loading time doesn't decrease further moving from 115200 to 460800? Is there some kind of rate limiter?

I had the same question. I'm guessing it's the rate limiting here.
https://github.com/PX4/Firmware/blob/e2b8fd7b118a232d5ffa406041acb58cde1c2543/src/modules/mavlink/mavlink_receiver.cpp#L2916
Plenty of room for future optimization...

@bkueng some measurements regarding CPU usage, on Pixhawk/fmu-v2:
master : baudrate:  57600, rate: 5000, mode: 0: 4.5%, 0.37%
master : baudrate:  57600, rate: 5000, mode: 2: 6.3%, 0.37%
master : baudrate:  57600, rate: 5000, mode: 8: 6.8%, 0.37%

this PR: baudrate:  57600, rate: 5000, mode: 0: 4.6%, 0.37%
this PR: baudrate:  57600, rate: 5000, mode: 2: 6.1%, 0.37%
this PR: baudrate:  57600, rate: 5000, mode: 8: 6.3%, 0.37%

master : baudrate: 460800, rate: 40000, mode: 0: 4.5%, 0.37%
master : baudrate: 460800, rate: 40000, mode: 2: 6.6%, 0.37%
master : baudrate: 460800, rate: 40000, mode: 8: 8.2%, 0.37%

this PR: baudrate: 460800, rate: 40000, mode: 0: 4.5%, 0.37%
this PR: baudrate: 460800, rate: 40000, mode: 2: 6.3%, 0.37%
this PR: baudrate: 460800, rate: 40000, mode: 8: 7.5%, 0.37%

It actually looks like the PR improves CPU...

@bkueng some measurements regarding CPU usage, on Pixhawk/fmu-v2:

It's the additional memory usage I'm concerned with.

@bkueng We can recuperate this with smaller UART TX buffers on some boards. I think we have the ~1K free RAM and if not this can be squeezed out where needed. This is a pretty significant concern overall and a constant issue for a long time.

@dagar I reviewed the diff and I saw the last commit message. I haven't reviewed the first commit message up until now, not expecting anything other than the usual Daniel commit messages. I'm on the fence if they should be reviewed or if we rather would like to just make sure that individual commit messages for PRs that are not flagged as WIP or draft should always be clean in wording.

@dagar I reviewed the diff and I saw the last commit message. I haven't reviewed the first commit message up until now, not expecting anything other than the usual Daniel commit messages. I'm on the fence if they should be reviewed or if we rather would like to just make sure that individual commit messages for PRs that are not flagged as WIP or draft should always be clean in wording.

My preference is to squash & merge by default unless the pull request is a sprawling change that touches many parts of the system and each incremental change was actually tested. That way we can keep the ugly history ("how the sausage was made") including review and test history and follow up changes. Then with squash & merge we get one last chance to verify the commit message actually reflects what the overall change evolved into and decide if it should reference the PR # or has enough information on its own.

@bkueng some measurements regarding CPU usage, on Pixhawk/fmu-v2:

It's the additional memory usage I'm concerned with.

Oops, I remembered that wrongly. But I do agree with the fact that this optimization potentially allows to make the tx buffers smaller.


feat: language support via locale config

This is an improvement for #69, this PR sets the language using the config command.
Examples:
Set Spanish

Set French

@privatenumber Can you take a look at this? I made the changes we discuss in the another PR, if you consider this is the right way to handle languages, I'll close the previous PR.

Thanks @vcgtz Looking forward to your changes.

Sorry for the delay and thanks!

@privatenumber Don't worry about it, it was my pleasure to contribute :+1:

Very useful feature , would you like to confirm that there will be a hint on how to use this feature in the readme?
like aicommits config set locale=<language code>
Or am I honored to help with this?


Full Module RNG Scope

Converting this to an issue since I'm repeatedly stumbling into the current non-streamlined implementation for module random-key management (see discussion below; it doesn't present a solution yet, but it writes down some ideas) .
The biggest issue that the current design has is that it prevents me from composing generic modules without having to awkardly pass a scope of RNGs (and likely make a mistake).
This snippet shows a example use-case for sampling and applying random functions,
from typing import Generic, TypeVar

import jax
import jax.numpy as jnp

from flax import struct
from flax import linen as nn

M = TypeVar('M', bound=nn.Module)

@struct.dataclass
class Composite(Generic[M]):
    module: M

    def do(self, key: jax.Array, ins: jax.Array):
        # Sample random module parameters, and apply the function.

        key_init, keys_sample = jax.random.split(key)

        # Problem: Throws an error if an arbitrary key in `M` is not covered.
        params = self.module.init(key, ins)

        # Problem: I do not know the RNG scope of self.module
        outs = self.module.apply(params, ins)

        return outs

c = Composite(nn.Dense(5))
c.do(jax.random.key(0), jnp.ones(5))
> Array([-0.7242578 ,  0.50101703,  0.44527686,  0.31400478, -0.02983775],      dtype=float32)

I'd personally suggest that flax should silently split up key inside the init and apply for me. Another solution is that flax returns all observed rngs in the Module scope.
Discussed in https://github.com/google/flax/discussions/3375

Originally posted by joeryjoery September 28, 2023
Hi, if I implement a Module that makes use of make_rng, e.g., Dropout, I now have to manually specify the rng when I use the Module's call function. Though I completely agree that the RNG streams should be split into distinct paths like it is now, it is quite annoying that I have to bookkeep this at every call to Module.apply.
import jax
import flax.linen as nn

rng = jax.random.PRNGKey(0)

class A(nn.Module):

    def __call__(self):
        x = self.make_rng('x')
        y = self.make_rng('y')

        return jax.random.uniform(x, ()) + jax.random.uniform(y, ())

myA = A()

# Currently
keys = jax.random.split(rng, 3)
scope = {'params', 'x', 'y'}  # I don't want to do this manually

# Desired functionality (or something like this)
# scope = myA.get_rng_scope()

rngs = dict(zip(scope, keys))

v = myA.init(rngs)
_ = myA.apply(v, rngs=rngs)

Is there already something like this implemented? Or could this be done in the future?
I'd also argue that flax should allow me to just pass in one global RNG key to either init or apply.
# Even better (just use one key)
_ = myA.apply(v, rngs=jax.random.PRNGKey(0))

This functionality would be great to have with the option to pass in the RNGS explicitly. This has no consequences for reproducibility since the rngs need to be split somewhere anyway.
At the moment the RNG design forces me to write work-arounds in my code which requires me to pass around a namespace that tracks all registered scopes. But this is quite error-prone and ideally this would be handled internally.

I'm not fully aware of the flax internals, I wasn't aware of scope.rngs so this is great to know! I can already use this to unit-test e.g., typos in creating rngs (e.g., I have the RNG stream dropout and somewhere else drpout).
Though in your code example you still have to pass in all keys explicitly in the init, which I might not know in advance for generic Modules and would have to explicitly trace.
It would be amazing if this would be handled internally.

After #3669, you can now pass in a single RNG key under the 'default' name to .init and .apply. For example {'default': jax.random.key(0)}

After #3669, you can now pass in a single RNG key under the 'default' name to .init and .apply. For example {'default': jax.random.key(0)}

Great, this is a much better alternative. Thanks a lot!


Receive Pane lack of scroll on word wrap

The Receive Pane now seems to fail to scroll when word wrap happens.  See the video for a demo.

UPDATE:  This is worse than first thought.  Bumping up priority.
Using this code, where there's no CR (carriage return) in the DEBUG statement, causes the scroll thumb to change but no automatic scrolling action on the Receive Pane.  This is incorrect behavior that was working correctly in v0.5.0.  This lack of CR was accidental in this code, but there are often times where users leave it off on purpose.
'{$STAMP BS2}
'{$PBASIC 2.5}

DO
  TOGGLE 0
  PAUSE 100
  DEBUG "Toggling: ", DEC W0
  W0 = W0 + 1
LOOP

Verified fixed.


Create a script that converts a CSV with prompts into an RKG

We cannot directly support CSV files because it's used exclusively for inputs, so a script to convert it from a CSV to an RKG in the repository would be nice to have. Users should supply a CSV, and if it's determined to be valid, the script should ask for metadata involving character, vehicle, etc.

The rkg header should be as minimal as possible, since we only care about the inputs and the following metadata:

Character
Vehicle
Manual/Automatic drift

Information like the mii/mii name and country are not relevant here correct?

Right. A more general CSV to RKG script would be nice to have for the TAS community, but for the purposes of this project we do not care about any of the extra information.

As we only allow have support for one test case at the moment, this will only reach relevancy alongside #8.


[RENDERER] no update of nested, conditional, templates

no template update, after property change, if an HTML string is returned instead of just a plain string (or literal)
https://codepen.io/KingDE/pen/YzmabbL?editors=1010
Originally posted by @KinG-DE in https://github.com/webtides/element-js/issues/123#issuecomment-2446261260

this was solved by the attached MR


SocketChannel: allow bind before register

Motivation:
In certain cases, on a client channel, the user might want to bind in
the channelInitializer which happens before the channel is registered
or active.
Modifications:
Allow bind in the channelInitializer.
Result:
Can bind before connect.

ping @Lukasa

I believe we have an outstanding question on this here.

@Lukasa apologies. So I agree we should think about the question over there and I think we should at least file an issue to resolve/think about it. On the other hand, this PR here adds functionality and doesn't make any working program worse. Therefore I'd argue that is an improvement. Sure, the new functionality is a bit brittle to use because you need to know how to not use happy eyeballs by resolving the address manually before.
Regarding your question, I'd say we should take option 1 (don't care) for now and in an issue that is to be filed we should decide if we want 2 or 3. WDYT?

I'm fine with choosing to not care. I'm not necessarily thrilled about it, but we can live with that outcome. We should file an issue to add support that is compatible with HE.

I think the comment here needs addressing.

I think the comment here needs addressing.

@Lukasa so you'd be happy to leave the HE wrinkle in here for now (creating an issue ofc) but you'd like me to add support for ServerSocketChannel and DatagramChannel?

No, my thing is that I think it does work now with the change you've made, but the tests don't explore the behaviour. I think they should.

@weissi why not expose a bind(...) method on the bootstrap itself ? This way the user can bind and the call connect(...) on the channel itself. ? Thats what we do in netty.

We don't want them to call connect on the channel itself because then they lose Happy Eyeballs. A proper Happy-Eyeballs aware implementation does have both a bind and connect method on the client bootstrap, but only so that HE knows what the local interface will be to discard a bunch of connection targets.

@Lukasa hmm... but isn't it quite error prone to call bind(...) in the initialiser as well as the user may end up trying to bind to the same port multiple times (as the initialiser may be run multiple times) ?

Yes, but that's no different to calling bind then connect except that they do wasted work. In both cases only one connection attempt can be made.
The right thing to do is to tell the resolver to restrain itself to only connections that can be made from a given local interface. getaddrinfo is capable of using that information to scope its results, as are many others. But the whole nature of this patch is that it doesn't do that.

@Lukasa what about we do bind on the bootstrap and disable HE if that has been done?

I'm fine with that. It's not quite as good, but it doesn't require an amendment to the Resolver interface.

@Lukasa exactly. It's a pretty rare use-case so I think that's okay. We can add an issue for NIO3. @normanmaurer WDYT?

I don't think it has to be NIO3: the in-planning "new" resolver interface can take a hints structure that requests that all provided addresses be reachable from a given interface. Then we can go back to firing bind through the pipeline for HE.

@Lukasa very good point!

@Lukasa is there a bug for the new resolver interface already? So we can add thoughts/requirements?

#1487

Closing in favour of #1490.


Ubuntu 20.04 and ROS Noetic -> ImportError: No module named rospkg

Hi, I'm working on Ubuntu 20.04 and ROS Noetic. I want to install tello_driver but I get the following error message:
SUMMARY
========

PARAMETERS
 * /rosdistro: noetic
 * /rosversion: 1.15.11
 * /tello/tello_driver_node/camera_calibration: /home/user...
 * /tello/tello_driver_node/connect_timeout_sec: 10.0
 * /tello/tello_driver_node/local_cmd_client_port: 8890
 * /tello/tello_driver_node/local_vid_server_port: 6038
 * /tello/tello_driver_node/stream_h264_video: True
 * /tello/tello_driver_node/tello_cmd_server_port: 8889
 * /tello/tello_driver_node/tello_ip: <IP_ADDRESS>

NODES
  /tello/
    image_compressed (image_transport/republish)
    tello_driver_node (tello_driver/tello_driver_node)

auto-starting new master
process[master]: started with pid [35315]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to b96e7966-fc1d-11eb-befe-89052bf0da48
process[rosout-1]: started with pid [35325]
started core service [/rosout]
process[tello/tello_driver_node-2]: started with pid [35332]
process[tello/image_compressed-3]: started with pid [35333]
Traceback (most recent call last):
  File "/home/user/catkin_ws/src/tello_driver/nodes/tello_driver_node", line 2, in <module>
    import rospy
  File "/opt/ros/noetic/lib/python3/dist-packages/rospy/__init__.py", line 49, in <module>
    from .client import spin, myargv, init_node, \
  File "/opt/ros/noetic/lib/python3/dist-packages/rospy/client.py", line 52, in <module>
    import roslib
  File "/opt/ros/noetic/lib/python3/dist-packages/roslib/__init__.py", line 50, in <module>
    from roslib.launcher import load_manifest  # noqa: F401
  File "/opt/ros/noetic/lib/python3/dist-packages/roslib/launcher.py", line 42, in <module>
    import rospkg
ImportError: No module named rospkg
[tello/tello_driver_node-2] process has died [pid 35332, exit code 1, cmd /home/user/catkin_ws/src/tello_driver/nodes/tello_driver_node __name:=tello_driver_node __log:=/home/user/.ros/log/b96e7966-fc1d-11eb-befe-89052bf0da48/tello-tello_driver_node-2.log].
log file: /home/user/.ros/log/b96e7966-fc1d-11eb-befe-89052bf0da48/tello-tello_driver_node-2*.log

I've seen that ImportError: No module named rospkg is pretty well known and I've been reading through a lot of posts but most of the people either have problems with conda (which does not run on my machine) or with the python version (they changed the version from 2. to 3.). My python version is 3.8.10 so I don't think that there is the problem. Does anyone know what the problem could be? Help would be much appreciated.

The command below worked for me.
sudo pip install rospkg

perhaps, your node tries to start python and cant find it. try to change first line in :
sudo nano /opt/ros/noetic/lib/tello_driver/tello_driver_node.py
to looks like:
#! /usr/bin/env python3

@zoldaten Thanks it worked struggled for 3 hours


New provider: Huawei Cloud DNS

Hello, we have completed the provider of Huawei Cloud DNS and applied to add it to libdns.
https://github.com/devhaozi/huaweicloud

Here is the screenshots from the test:

Great, thank you! I've sent an email inviting you to push to the new repositories, including a caddy-dns repository for use with Caddy.


(issue #170 ) Moore's Voting Algorithm

Moore's Voting Algorithm is an efficient algorithm used to find the majority element in an array, which is defined as an element that appears more than half the time in the collection. The algorithm operates in linear time 𝑂(𝑛) and uses constant space  𝑂(1), making it optimal for scenarios where space efficiency is critical. It works by maintaining a candidate for the majority element and a counter to track its occurrence. If the counter reaches zero, a new candidate is selected. Finally, the algorithm verifies whether the candidate is indeed the majority element. This approach is particularly useful in data streams and real-time applications, providing a reliable means of determining prevalent elements in large datasets.
Proof of working results of algo :

@pankaj-bind Please assign me the labels level 3, hacktoberfest. gssoc-extd and allow the merge
i hope youll review and consider it.

@pankaj-bind Please assign me the labels level 3, hacktoberfest. gssoc-extd and allow the merge i hope youll review and consider it.

@pankaj-bind hello please look into this !

@pankaj-bind please allow to merge

🎉 Thank you for your contribution! Your pull request has been submitted successfully. A maintainer will review it as soon as possible. We appreciate your support in making this project better

please mention the maintainer to review and assign me the level 2gssoc-extd hactoberfest labels and merge this


Uncaught initial bisect points must have opposite signs

[
{sets:['行业兴趣人群'],size:323697564},
{sets:['本品了解'],size:203382183},
{sets:['本品吸引'],size:195035862},
{sets:['本品问询'],size:63038111},
{sets:['本品了解','行业兴趣人群'],size:56497283},
{sets:['本品吸引','行业兴趣人群'],size:125644736},
{sets:['本品问询','行业兴趣人群'],size:63038111}
]

{sets:['本品问询'],size:63038111},
{sets:['本品问询','行业兴趣人群'],size:63038111}
two sets is equal


Features: Use case 2d

This becomes use case 2d.
TODO

[ ] total system cost: add invest costs

Feature starts with https://github.com/modex-flexmex/oemo-flex/pull/47/commits/aac106ed5138e18a316cbb0f8646f1b9bbc23fdd

Because of the dev merge this already uses input v0.04. experiment_1/001_data_raw/Data/In/v0.04/Scalars.csv needs to be added!

Postprocessing is a bit hasty. It tries to read its template file before any processing, only by importing it. Getting:
FileNotFoundError: [Errno 2] File /home/unndreay/Workspaces/oemo-flex/experiment_1/004_results_data_template/v0.04/Template/Scalars.csv does not exist: '/home/unndreay/Workspaces/oemo-flex/experiment_1/004_results_data_template/v0.04/Template/Scalars.csv'
Should happen not before its main() has been called.

The whole input directory structure need to be added:
FileNotFoundError: [Errno 2] No such file or directory: '/home/unndreay/Workspaces/oemo-flex/experiment_1/001_data_raw/Data/In/v0.04/Energy/FinalEnergy/Electricity'

The comparison call
https://github.com/modex-flexmex/oemo-flex/blob/86101b5dbdf03d8b1c594aeb63957e68bf2efb6a/experiment_1/scripts/FlexMex1_2d/preprocessing.py#L76
needs to be commented out in the first run to generate a preprocessing default result.

All right. Only busses.csv changed: there is also a 'gas' bus now.

Scalars.csv output parameters
Transmission_Capacity_Electricity_Grid
Transmission_FixOM_Electricity_Grid
Transmission_Invest_Grid

are missing, since electricity-transmission is not expandable yet. And not planned to be since needs a bigger change.

Merged with dev and re-run. Nothing changed except from system cost as expected. Ready to be merged.

I get the same on my computer. Ready to merge.


'When done' causing wrong done date

Please check that this issue hasn't been reported before.

[X] I searched previous Bug Reports didn't find any similar reports.

Expected Behavior
When I check the below task as done by clicking the check field, it should add today's done date (2024-05-31) and create the recurring task.
- [ ] 🦷 Dentist Check up Appointment 🔁 every 6 months when done ⏳ 2024-05-31
which would turn into:
- [x] 🦷 Dentist Check up Appointment 🔁 every 6 months when done ⏳ 2024-05-31 ✅ 2024-05-31
- [ ] 🦷 Dentist Check up Appointment 🔁 every 6 months when done ➕ 2024-05-31 ⏳ 2024-11-30
Current behaviour
When checked, the task adds yesterday as it's completion date instead.
- [x] 🦷 Dentist Check up Appointment 🔁 every 6 months when done ⏳ 2024-05-31 ✅ 2024-05-30
- [ ] 🦷 Dentist Check up Appointment 🔁 every 6 months when done ➕ 2024-05-31 ⏳ 2024-11-30
I have tried it without the 'when done' words in the recurrence rule and it works correctly.
Steps to reproduce

I have been able to reproduce this in sandbox vault with only Tasks installed

Maintainer's note: See https://github.com/obsidian-tasks-group/obsidian-tasks/issues/2867#issuecomment-2192679323 for reproduction steps.
Which Operating Systems are you using?

[ ] Android
[ ] iPhone/iPad
[ ] Linux
[ ] macOS
[X] Windows

Obsidian Version
1.6.1 (Insider Release), Installer version 1.5.3
Tasks Plugin Version
7.3.0
Checks

[X] I have tried it with all other plugins disabled and the error still occurs

Possible solution
No response

What time zone was the computer/device set to?
What time of day did you observe the problem?

First thought: I am wondering whether this might be the underlying cause:

https://github.com/obsidian-tasks-group/obsidian-tasks/issues/2866

Perhaps some calculation involved in the ‘when done’ code is unintentionally editing a date value.
A test would need to written to demonstrate the problem behaviour - hence asking for time and time zone info.

What time zone was the computer/device set to?
What time of day did you observe the problem?

I am in +0800 timezone (Perth/Australia) and it was about 2pm in the afternoon.
Trying it in a sandbox vault today (2024-06-01, ~10.30am, same timezone above) and apparently it has decided to work correctly.
I wonder if there is a problematic connection between the Recurrence Date of the newly created task and the Completion date of the old task?

Thanks, I did wonder... It seems to be the only time-zone that features regularly...

#2309
#1200

Steps to reproduce

I have been able to reproduce this in sandbox vault with only Tasks installed

For future reference, it really helps with development if there are actual steps listed to reproduce the bug, instead of having to gather information from other parts of the ticket....

Steps to reproduce

Install and enable the Tasks plugin (tested with Tasks 7.5.0)
Edit the Tasks settings:

Enable 'Next recurrence appears on the line below'

Paste the following line in to any Markdown note:

- [ ] #task 🦷 Dentist Check up Appointment ⏳ 2024-05-31 🔁 every 6 months when done

Put the cursor in that line and run Tasks: Create or edit task

Change the status to Done
Paste 2024-05-31 in to the Done date field
Click Apply

Expected result:

[x] #task 🦷 Dentist Check up Appointment 🔁 every 6 months when done ⏳ 2024-05-31 ✅ 2024-05-31
[ ] #task 🦷 Dentist Check up Appointment 🔁 every 6 months when done ⏳ 2024-11-30

The Done date should be ✅ 2024-05-31, which is the supplied value.
Actual result:

[x] #task 🦷 Dentist Check up Appointment 🔁 every 6 months when done ⏳ 2024-05-31 ✅ 2024-05-30
[ ] #task 🦷 Dentist Check up Appointment 🔁 every 6 months when done ⏳ 2024-11-30

The Done date is a day earlier than the specified value.
Note:
Repeating with this line gives the correct result:
- [ ] #task 🦷 Dentist Check up Appointment ⏳ 2024-05-31 🔁 every 6 months

Removing the when done gives the correct result in this case.

Thanks, I did wonder... It seems to be the only time-zone that features regularly...

Completing a daily recurring task creates task set to same day [the day before Australian clocks switch to daylight savings] #2309
Recurring Task 'every day' gets stuck at the end of a month [the day before Australian clocks switch to daylight savings] #1200

It doesn't depend on timezone after all...
I can reproduce it in UTC and Current UK (BST) time zones.
I am now pretty sure the cause is an instance of this:

https://github.com/obsidian-tasks-group/obsidian-tasks/issues/2866

Hi @Azurewren, huge thanks for reporting this!
It will be fixed in the next release....


'each' resource is a noop

https://github.com/xFAANG/askql/blob/master/src/askvm/resources/core/each.ts
export const each = resource({
  type: any,
  resolver: async () => null,
});

Hi @Fifciu , would you like to take a look at this issue? :)

@czerwinskilukasz1 what is the expected behaviour of each?

Right.
My proposal is the following:
each should accept 2 arguments: an array and a function, which should be applied on each of the elements of the array.
Function signature should be: fun(t: char, index: number, firstArrayElement: boolean, lastArrayElement: boolean):any.
The result values are disregarded.
Example:
  const lorem = 'lorem ipsum dolor sit amet consectetur adipiscing elit'
  lorem:split(' '):each( fun(t: char, index: number, firstArrayElement: boolean, lastArrayElement: boolean) {
	  const cap = t:toUpperCase
    if (last) {
      result = result + cap
    } else {
      result = result + concat(cap, ' ')
    }
  })

@mhagmajer , any thoughts?

Why first and last?
Why not just do like the JS forEach function?
(item: any, index: number, array: any[]) => {}

I like @YonatanKra proposal of following the JS signature

Me too.


Improve references

Codecov Report

Merging #761 into master will increase coverage by 0.1%.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           master    #761     +/-   ##
========================================
+ Coverage      69%   69.1%   +0.1%
========================================
  Files         292     292
  Lines        7656    7656
========================================
+ Hits         5283    5291      +8
+ Misses       2373    2365      -8

Impacted Files
Coverage Δ

src/article/converter/bib/BibConversion.js
0% <ø> (ø)
:arrow_up:

src/article/InternalArticleSchema.js
95.45% <ø> (ø)
:arrow_up:

src/article/ArticleConstants.js
100% <ø> (ø)
:arrow_up:

src/article/ExperimentalArticleValidator.js
90.32% <0%> (+1.61%)
:arrow_up:

src/article/ArticleAPI.js
43.57% <0%> (+1.67%)
:arrow_up:

src/article/shared/NodeModelComponent.js
90% <0%> (+2%)
:arrow_up:

src/kit/ui/FormRowComponent.js
100% <0%> (+20%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3fb0910...e0e67cc. Read the comment docs.


[ADAP-551] [Bug] Datetime incremental tables scanning more of table than expected

Is this a new bug in dbt-bigquery?

[X] I believe this is a new bug in dbt-bigquery
[X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior
I have a table partitioned by day on a datetime column. When running an incremental run BigQuery performs a full table scan instead of an optimized partition scan.
This is likely due to a BigQuery bug where the function datetime_trunc ignores partitions.
Example:
-- `events` is partitioned by day on the datetime column `created_at`
select * from events where datetime_trunc(created_at, DAY) in (current_date) -- 650GB
select * from events where timestamp_trunc(created_at, DAY) in (current_date) -- 100MB
select * from events where date(created_at) in (current_date) -- 100MB

Expected Behavior
Optimized partition scan with 100mb usage but I'm seeing a full table scan with 650gb usage.
Steps To Reproduce
The events table is partitioned by day on a datetime column with the following config:
{{ config(
    materialized='incremental',
    unique_key='id',
    partition_by={
      "field": "created_at",
      "data_type": "datetime"
    },
    incremental_strategy = 'insert_overwrite',
    partitions = [ 'current_date', 'date_sub(current_date, interval 1 day)' ]
  )
}}

Running an incremental sync will use datetime_trunc resulting in a full table scan. Here is an extract from the run code:
    merge into `zipline-datawarehouse`.`analytics`.`events` as DBT_INTERNAL_DEST
        using (
  --- SQL MODEL CODE
) as DBT_INTERNAL_SOURCE
        on FALSE
    when not matched by source
         and datetime_trunc(DBT_INTERNAL_DEST.created_at, day) in (
              current_date, date_sub(current_date, interval 1 day)
          )
        then delete
    when not matched then insert
        (COLUMNS)
    values
        (COLUMNS)

Relevant log output
No response
Environment
- OS: MacOS
- Python: 3.10.9
- dbt-core: 1.5.0
- dbt-bigquery: 1.5.0

Additional Context
We can work around this issue by setting the partition_by.data_type to timestamp. dbt will happily use timestamp_trunc and BigQuery will only look up the necessary partitions. (Note that this was working in v1.2, it stopped working in v1.4, and it's now working again in v 1.5).
{{ config(
    materialized='incremental',
    unique_key='id',
    partition_by={
      "field": "created_at",
      "data_type": "timestamp"
    },
    incremental_strategy = 'insert_overwrite',
    partitions = [ 'current_date', 'date_sub(current_date, interval 1 day)' ]
  )
}}

A similar issue with date partitioning but it doesn't seem to be the same cause: https://github.com/dbt-labs/dbt-bigquery/issues/708
A mention of the bug where datetime_trunc makes BigQuery perform full table scans: https://github.com/dbt-labs/dbt-utils/issues/393

Thanks for reaching out @pcreux !
I'm not a seeing a difference between the config you reported in the "Steps To Reproduce" vs. the workout in the "Additional Context" section.
Is it as simple as this, or is there more involved?
- "data_type": "datetime"
+ "data_type": "timestamp"

Could you take a peek and update as needed?

Hey @dbeatty10 ! You are right, it's indeed "data_type": "timestamp".  I just updated the code snippet.

This is likely due to a BigQuery bug where the function datetime_trunc ignores partitions.

@pcreux do you have a link to this bug report on the Google side of things? It will be good to hear what Google says about it.
If their guidance is to just use timestamp_trunc instead of datetime_trunc when the underlying data type of the partitioned column is datetime, then the fix would fairly simple here:
https://github.com/dbt-labs/dbt-bigquery/blob/495566a24e9e8a1f35ee366ae1ecf8e944a6f966/dbt/adapters/bigquery/impl.py#L123-L126
Probably something like:
        if self.data_type_should_be_truncated() and self.data_type == "datetime":
            return f"timestamp_trunc({column}, {self.granularity})"
        elif self.data_type_should_be_truncated():
            return f"{self.data_type}_trunc({column}, {self.granularity})"
        else:
            return column

I had a few changes to solve issues with trunc functions in https://github.com/dbt-labs/dbt-bigquery/commit/7c216445f8009baa9cec4d61dd56693be1dd79fa
Could you try 1.6.X version to see if it works better for you?


[BUG] libcamera-hello doesn't show preview on console

Describe the bug
libcamera-hello doesn't show preview on console
Bug report
2021-10-30-raspios-bullseye-armhf-lite  (so no gui/X)
libcamera-hello - no image and a message: "Preview window unavailable"
enable legacy,  raspistill will show  an image

Just so we can be sure what you're running, can you please report the output of:

uname -a
vcgencmd version
libcamera-hello --version

It sounds like you have a keyboard and screen plugged directly into your Pi, and you're logged in directly rather than through ssh. Can you confirm that?
Running without X WIndows, you should get the DRM/KMS-based preview window automatically. Could you check your /boot/config.txt contains dtovleray=vc4-kms-v3d. Actually pasting the whole thing in here would be helpful. Thanks!

gah - my bad.  I was using an old config.txt that did not have dtovleray=vc4-kms-v3d
using the stock settings, I get the DRM/KMS-based preview.
sorry!


Fix mobile navbar dropdowns

#342  🔗

✍ Description

Add dropdown data target to Avatar which enables clicking instead of hover on mobile
Navbar collapse menu checks if user is logged in before displaying login/adopt menu items
Add tests for mobile screen size login/logout

📷 Screenshots/Demos

If you end up accepting this, please take an extra moment to review. I am very new to all of this!!

Please run rails standard:fix

Hey @jmilljr24 thank you for taking the initiative! Much appreciated.
So you are right in noticing that we had discussion around the use of ViewComponent, and that instead we would prefer Nice Partials. So, I won't approve this for now as I am reverting this PR https://github.com/rubyforgood/pet-rescue/pull/326 which your PR refers to. It would be great if you want to do the same changes when the new PR is up to add avatars with nice_partials?
In the meantime, feel free to pick up another issue :)

Sounds good! I'll keep an eye out and see what I can do.


Net::OpenTimeout error when fetching the blog posts

here is my backtrace:
08:03:07  /usr/lib/ruby/2.5.0/net/protocol.rb:41:in `ssl_socket_connect': Net::OpenTimeout (Net::OpenTimeout)
08:03:07  	from /usr/lib/ruby/2.5.0/net/http.rb:985:in `connect'
08:03:07  	from /usr/lib/ruby/2.5.0/net/http.rb:920:in `do_start'
08:03:07  	from /usr/lib/ruby/2.5.0/net/http.rb:909:in `start'
08:03:07  	from /usr/lib/ruby/2.5.0/net/http.rb:1458:in `request'
08:03:07  	from /home/user/workspace/Website_planet-kde-org/vendor/ruby/2.5.0/gems/fetcher-0.4.5/lib/fetcher/worker.rb:203:in `get_response'
08:03:07  	from /home/user/workspace/Website_planet-kde-org/vendor/ruby/2.5.0/gems/fetcher-0.4.5/lib/fetcher/worker.rb:56:in `get'
08:03:07  	from /home/user/workspace/Website_planet-kde-org/vendor/ruby/2.5.0/gems/pluto-feedfetcher-0.1.5/lib/pluto/feedfetcher/cond_get_with_cache.rb:34:in `fetch'
08:03:07  	from /home/user/workspace/Website_planet-kde-org/vendor/ruby/2.5.0/gems/pluto-update-1.6.3/lib/pluto/update/feed_refresher.rb:57:in `refresh_feed_worker'
08:03:07  	from /home/user/workspace/Website_planet-kde-org/vendor/ruby/2.5.0/gems/pluto-update-1.6.3/lib/pluto/update/feed_refresher.rb:49:in `block in refresh_feeds_for'
08:03:07  	from /home/user/workspace/Website_planet-kde-org/vendor/ruby/2.5.0/gems/activerecord-<IP_ADDRESS>/lib/active_record/relation/delegation.rb:85:in `each'
08:03:07  	from /home/user/workspace/Website_planet-kde-org/vendor/ruby/2.5.0/gems/activerecord-<IP_ADDRESS>/lib/active_record/relation/delegation.rb:85:in `each'
08:03:07  	from /home/user/workspace/Website_planet-kde-org/vendor/ruby/2.5.0/gems/pluto-update-1.6.3/lib/pluto/update/feed_refresher.rb:48:in `refresh_feeds_for'
08:03:07  	from /home/user/workspace/Website_planet-kde-org/vendor/ruby/2.5.0/gems/pluto-1.3.4/lib/pluto/cli/updater.rb:51:in `update_for'
08:03:07  	from /home/user/workspace/Website_planet-kde-org/vendor/ruby/2.5.0/gems/pluto-1.3.4/lib/pluto/cli/updater.rb:31:in `run'
08:03:07  	from /home/user/workspace/Website_planet-kde-org/vendor/ruby/2.5.0/gems/pluto-1.3.4/lib/pluto/cli/main.rb:232:in `block (3 levels) in <top (required)>'
08:03:07  	from /home/user/workspace/Website_planet-kde-org/vendor/ruby/2.5.0/gems/pluto-1.3.4/lib/pluto/cli/main.rb:208:in `each'
08:03:07  	from /home/user/workspace/Website_planet-kde-org/vendor/ruby/2.5.0/gems/pluto-1.3.4/lib/pluto/cli/main.rb:208:in `block (2 levels) in <top (required)>'
08:03:07  	from /home/user/workspace/Website_planet-kde-org/vendor/ruby/2.5.0/gems/gli-2.19.0/lib/gli/command_support.rb:131:in `execute'
08:03:07  	from /home/user/workspace/Website_planet-kde-org/vendor/ruby/2.5.0/gems/gli-2.19.0/lib/gli/app_support.rb:296:in `block in call_command'
08:03:07  	from /home/user/workspace/Website_planet-kde-org/vendor/ruby/2.5.0/gems/gli-2.19.0/lib/gli/app_support.rb:309:in `call_command'
08:03:07  	from /home/user/workspace/Website_planet-kde-org/vendor/ruby/2.5.0/gems/gli-2.19.0/lib/gli/app_support.rb:83:in `run'
08:03:07  	from /home/user/workspace/Website_planet-kde-org/vendor/ruby/2.5.0/gems/pluto-1.3.4/lib/pluto/cli/main.rb:390:in `<top (required)>'
08:03:07  	from /home/user/workspace/Website_planet-kde-org/vendor/ruby/2.5.0/gems/activesupport-<IP_ADDRESS>/lib/active_support/dependencies.rb:325:in `require'
08:03:07  	from /home/user/workspace/Website_planet-kde-org/vendor/ruby/2.5.0/gems/activesupport-<IP_ADDRESS>/lib/active_support/dependencies.rb:325:in `block in require'
08:03:07  	from /home/user/workspace/Website_planet-kde-org/vendor/ruby/2.5.0/gems/activesupport-<IP_ADDRESS>/lib/active_support/dependencies.rb:291:in `load_dependency'
08:03:07  	from /home/user/workspace/Website_planet-kde-org/vendor/ruby/2.5.0/gems/activesupport-<IP_ADDRESS>/lib/active_support/dependencies.rb:325:in `require'
08:03:07  	from /home/user/workspace/Website_planet-kde-org/vendor/ruby/2.5.0/gems/pluto-1.3.4/lib/pluto.rb:24:in `main'
08:03:07  	from /home/user/workspace/Website_planet-kde-org/vendor/ruby/2.5.0/gems/pluto-1.3.4/bin/pluto:5:in `<top (required)>'
08:03:07  	from /home/user/workspace/Website_planet-kde-org/vendor/ruby/2.5.0/bin/pluto:23:in `load'
08:03:07  	from /home/user/workspace/Website_planet-kde-org/vendor/ruby/2.5.0/bin/pluto:23:in `<main>'

Thanks for reporting the error when fetching blog post.  Is this a "sporadic" error e.g. only happens once in a while  or is it always and every time?   If you can add the --verbose switch on the command line it should tell you more e.g. is it the first web feed or which one is causing the error?
Note: due to Corona COVID-19 I'm currently mostly offline (and sorry don't expect fast response) - thanks for your patience. stay healthy,

I cannot reproduce with my test feeds but it looks like Net::OpenTimeout is another unhandled exception - I added Net::OpenTimeout to the rescue clause and it should get handled and logger but not abort. Please, update to the latest pluto-feedfetcher gem (v.0.1.6), see https://rubygems.org/gems/pluto-feedfetcher.

Thanks a lot, the new version now run on our server. :heart_eyes:  I will report if the problem reappear.


Transpilation of ES6 imports is inconsistent with babel

Currently, babel and typescript handle transpilation of ES6 imports differently.
The following works in typescript:
import * as rp from 'request-promise';

rp('http://github.com').then(console.log);

but I get an error when running it with babel's transpilation:
TypeError: object is not a function
    at Object.<anonymous> (./test.js:7:1)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Function.Module.runMain (module.js:497:10)
    at startup (node.js:119:16)
    at node.js:906:3

To get it to work with babel, I have to do the following:
import rp from 'request-promise';

but that returns an error when compiling with typescript:
test.ts(1,8): error TS1192: Module ''request-promise'' has no default export.

I currently have a pipeline of typescript --tsc--> es6 --babel--> es5 so that I can use async/await while I wait for TypeScript to support it for ES5. However, that means I have to modify all my DefinitelyTyped declarations to have a default export so that tsc doesn't complain.
Discussions of which standard is better for transpilation put aside, my feeling is that TypeScript should follow the standard set by babel rather than using their own since babel is further along at supporting es6, and has a larger community at this point:
npm babel
33,936 downloads in the last day
174,998 downloads in the last week
551,455 downloads in the last month

npm typescript
15,303 downloads in the last day
101,148 downloads in the last week
357,086 downloads in the last month

In addition to keeping in step with the larger community, it would help adoption of typescript by making the transition to typescript from babel more seamless.

@eggers I ran into this same problem today, using the same sort of build pipeline. I ended up with the same sort of solution as you - changing typing files to have an export default.
In your first case though, you can also do something like this if you don't want to modify your typings. I'm not sure if you'll be able to have typing support for rp though this way:
import * as _rp from 'request-promise';
const rp = (_rp as any).default;
rp('http://github.com').then(console.log);

@JKillian
Yeah, man that is so ugly. I also don't like the solution because when typescript does finally support async/await --> ES5, and I drop babel from my pipeline, it'd break my code.
Also, there are even some other issues with typing support for request-promise because the function can take an Options parameter that has all optional properties, meaning the following is valid typescript:
rp(666).then(console.log);

I put more details here: borisyankov/DefinitelyTyped#6489

this looks like a typing issue. if request-promise has a default export it should be in the .d.ts file.

There's already an export = export present in the typings, but that has to be changed to work correctly with TS and Babel

@mhegazy The declaration is accurate for how typescript handles the transpilation, but the issue is that babel handles transpilation differently, and when I go ts-->es6-->es5 for babel's async/await support, i have to make declaration changes.

Looks like this will be handled by https://github.com/Microsoft/TypeScript/pull/5577

see https://github.com/Microsoft/TypeScript/issues/5285 for more details.


Private key exposure vulnerability in Ironclad's DSA signature implementation

The experimental IRONCLAD:SIGN-MESSAGE's DSA method uses CL:RANDOM to generate the message secret. Since CL:RANDOM is not cryptographically secure (on SBCL, it's based on the Mersenne Twister, which is highly insecure for crypto applications), it's possible that every message signature to leak a few bits of the private DSA key.
To fix this, some sort of cryptographically-secure random number source should be used--either an OS-specific source, the Fortuna implementation or another.
After it's fixed, all Ironclad users should change their DSA keypairs.

This was resolved in 60c37e8bdccdaac1c430d1a2982f1df5129e2944.


noetic release missing

https://github.com/ros-visualization/rqt/pull/241 was merged, but the package was not newly released into noetic. So the package is still broken in noetic and requires users to build from source... :(
Could you please trigger a quick release to get this fixed?

Sorry for the delay. We overlooked this issue. I've made a new release for Noetic: https://github.com/ros/rosdistro/pull/32529

thanks!


Add PagerDuty Alerting

Would be nice if PagerDuty support was added out of the box for ElastAlert.  I am already using the PagerDuty API with ElastAlert so I would be happy to work on this issue @Qmando if you would accept a pull request.

Hi @jasonrhaas I was just working also on that #290

This is now merged!


Add a new function to validate a device identify request.

Add a new Weave error WEAVE_ERROR_INVALID_DEVICE_DESCRIPTOR.

What are the appropriate unit / functional tests to create to validate this new functionality?

@gerickson : The error handling case introduced by this change can only be provoked manually or in the context of Happy tests that script the Weave Device Manager.  I have confirmed the correct behavior via manual testing.  However, currently there is no infrastructure in the Happy test suites for testing argument handling in the Device Manager.  Given the priority of this issue, I suggest we accept this change as-is.

Codecov Report

Merging #347 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #347   +/-   ##
=======================================
  Coverage   54.38%   54.38%
=======================================
  Files         341      341
  Lines       58177    58177
=======================================
  Hits        31639    31639
  Misses      26538    26538

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1e48f1a...a51f844. Read the comment docs.


[Update] Refactor useWalletNFTs hook and NFT handling

#4374  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @jnsdls and the rest of your teammates on  Graphite


Get-EnterPrisePKIHealthStatus.ps1 - Exception calling "SendRequest" with "0" argument(s): "Input data is not valid ASN-encoded GENERALIZED TIME."

Hi there,
Runing PSPKI <IP_ADDRESS> on Powershell 4.0 I am faced with a problem in the above script line 456 - $resp = $req.SendRequest(). To find the problem I commented out the try catch block around it. Once the method is called I get this:
Exception calling "SendRequest" with "0" argument(s): "Input data is not valid ASN-encoded GENERALIZED TIME."
I've seen this exception here: https://github.com/Crypt32/PSPKI/issues/55 but this time the conversion seems to happen within the library, which I'm not able to recompile.
Thx for helping me out.

I will take a look. Unfortunately, I’m very busy on other duties, but will try to check this by the end of week. It woukd be great if you could post a stack trace:
$error[0].exception.innerexception.stacktrace

Hi, thx for looking into this. Here is the stacktrace:
at SysadminsLV.Asn1Parser.Asn1Utils.DecodeGeneralizedTime(Byte[] rawData)
at PKI.OCSP.OCSPResponse.decodeTbsResponse(Asn1Reader tbsResponseData)
at PKI.OCSP.OCSPResponse.verifyAll(Asn1Reader tbsResponseData, Byte[] signature, Oid signatureAlgorithm)
at PKI.OCSP.OCSPRequest.SendRequest()
at CallSite.Target(Closure , CallSite , Object )

At first look it sounds like that OCSP responds with invalid data.
Exception occurs here: https://github.com/Crypt32/pkix.net/blob/84371e99162c91b5aba094c961aab0854d4a6c2e/PKI/OCSP/OCSPResponse.cs#L227
ProducedAt = Asn1Utils.DecodeGeneralizedTime(tbsResponseData.GetTagRawData());
According to RFC6960:
ResponseData ::= SEQUENCE {
   version             [0] EXPLICIT Version DEFAULT v1,
   responderID             ResponderID,
   producedAt              GeneralizedTime,
   responses               SEQUENCE OF SingleResponse,
   responseExtensions  [1] EXPLICIT Extensions OPTIONAL }

producedAt field is Generalized time, so I'm trying to decode it here: https://github.com/Crypt32/Asn1DerParser.NET/blob/49710a3788491aee4e45c3a7a73c6983227cfcca/Asn1Parser/Asn1Utils.cs#L250 and validate input type here: https://github.com/Crypt32/Asn1DerParser.NET/blob/49710a3788491aee4e45c3a7a73c6983227cfcca/Asn1Parser/Asn1Utils.cs#L253
if (asn.Tag != (Byte)Asn1Type.GeneralizedTime) {
    throw new InvalidDataException("Input data is not valid ASN-encoded GENERALIZED TIME.");
}

and input data is not generalized time I'm expecting, therefore throwing an exception you see.
I will check again if the code behaves as expected. If my tests are ok, then we may need to investigate what exactly returns OCSP server in your environment.

OK, any idea how I can grap the response, before the exception occurs?

Here is a code snippet you can use:
$req = New-Object pki.ocsp.ocsprequest "path to CA-issued cert"
$req.Encode()
$wc = New-Object net.webclient
$wc.Headers.Add("Content-Type", "application/ocsp-request");
$wc.Headers.Add("Accept", "*/*");
$wc.Headers.Add("User-Agent", "PowerShell PKI Module");
$wc.Headers.Add("Cache-Control", "no-cache");
$wc.Headers.Add("Pragma", "no-cache");
$url = $req.URL.OriginalString.Replace("`0",$null)
if ($url.EndsWith("/")) {
    $url += [uri]::EscapeUriString([convert]::ToBase64String($req.RawData)).Replace("+", "%2b").Replace("/", "%2f").Replace("=", "%3d")
} else {
    $url += "/" + [Uri]::EscapeUriString([convert]::ToBase64String($req.RawData)).Replace("+", "%2b").Replace("/", "%2f").Replace("=", "%3d")
}
$bytes = $wc.DownloadData($url)
$response = [convert]::ToBase64String($bytes)

$response will contain a base64 string of raw response. I will need this data.
do not forget to specify a path to a CA-issued certificate (for example, CA Exchange certificate)

Here is the response string:
MIIFyAoBAKCCBcEwggW9BgkrBgEFBQcwAQEEggWuMIIFqjCBoaIWBBR1En9TmbzWA2ZPCgRv08cGFOEy5BgPMjAxODExMjgwNDAxMDlaMHYwdDBMMAkGBSsOAwIaBQAEFNBey+wBfhgKg4JhBYyk9qIr9as5BBQcrbKPVb6s/CVTugDQAm4s6aENLgITKAAAAriC4Gyr3g9TYAAAAAACuIAAGA8yMDE4MTEyNzA0NTAwMVqgERgPMjAxODEy MDMyMzA4NDdaMA0GCSqGSIb3DQEBCwUAA4IBAQBAepKtn4PdFzzyNUz9C3Ap+nWX6vjI+MKsfPV+ybqb8zlDXbv2tNCUOlukhm2XLvLD041tikPKoQsCOhhAt/Mn7lK2wGq8COO06kC3fek6AH5ZPamatPmDJhuMKRYBwmkVkYuBBB+3XHiCPs54gfj9j8vvrvxHv0lQyC0u+AEWUqk31eLIXeVU4QNtrjeRX6EzVkOQFcrtM9a7e30TCa3m d40/jl5qwPFRt1dc6gEbTA4aGg7/lF5kgI57Qop3VMCLyHDNIuA/FhR23wFVK1z2OEDOOBftBnB3hEshz8YuAciozqxPNJ3o8AlAOCPpxmHusWHyXZF/lyc1jQLejQcBoIID7jCCA+owggPmMIICzqADAgECAhMoAAACvuwJHtj2TAmVAAAAAAK+MA0GCSqGSIb3DQEBCwUAMEYxCzAJBgNVBAYTAkRFMRQwEgYDVQQKEwtTaGFycCBFVSBR UzEhMB8GA1UEAxMYU2hhcnAgRXVyb3BlIFN1YiBDQSAzIFFTMB4XDTE4MTExOTIzMDg0N1oXDTE4MTIwMzIzMDg0N1owHjEcMBoGA1UEAxMTY3BraXEyMi5TaGFycFFTLlFkczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK7gXrFT/JnKGf3H0F2eP4Kr2f2r5E7JVEyi0B1TpbLfvr7xRl7zZb57dM/oghlBihkJ6unQJdCS f0hH1iLc6aFD1QWRUkdl6oHTKeou9BbqE83IrI9uxiEuxcf4zrCR7eFQyLHpaQmv5SgDM+2IbvS1xpaD64iZygnsK/TQFahUjM2lYjDIl3etMczpQooetHqmWJgTaYTnvLLNWWkIFs2fqLWGqDfIffx/49OyXs2r50jD+ehfT1SHWcx5RRSCtaJ3VGAL11jOlK0OjQaWak3RjeKec8SDrPkYI0Dr7wesNctEBhczI7Jals9HbIdGeJHfE015 EDhS8EZXyW4UFSMCAwEAAaOB9DCB8TA8BgkrBgEEAYI3FQcELzAtBiUrBgEEAYI3FQiEndkOhJfMXYLtjTfz6xqysCqBIIHji2aGlvpgAgFkAgEFMBMGA1UdJQQMMAoGCCsGAQUFBwMJMA4GA1UdDwEB/wQEAwIHgDAbBgkrBgEEAYI3FQoEDjAMMAoGCCsGAQUFBwMJMA8GCSsGAQUFBzABBQQCBQAwHwYDVR0jBBgwFoAUHK2yj1W+rPwl U7oA0AJuLOmhDS4wHQYDVR0OBBYEFHUSf1OZvNYDZk8KBG/TxwYU4TLkMB4GA1UdEQQXMBWCE2Nwa2lxMjIuU2hhcnBRUy5RZHMwDQYJKoZIhvcNAQELBQADggEBAKpcQpnf10KW2NoyvRZag+qdHJMGwTm9pwc+mLKvWqw9tvBjbwLgLza/Ifk3qG33dSsetw6g6kpfd0vFMGfSlcYPfqYwTFdEmskJXrwnrX17gIJactOuKIL/AgXKqjdK Y1TejxCXgRjq0pRAtdG2cAvWkaNi/DYbJXbQHwhQME1iDSpCwwrsnu3WTnpyE+rFKcyfQzWFmcxmeLrlnKRrY5B1IgKNuVY4GDJ9XPZpITt0P/tTKX2uye8oZm9QohA2zyLT+cNN7Ps7TI2OeTWHzmc/2SJGYBeoAyH9O4rYiK8gx00OR126AaAtsR+IxVE881qEe9GZ7hWfwdqxd6gfXKU=

Ok, I found the issue. This is an issue with my ASN.1 binary parser. This will require a hard work from my side in order to make a reliable fix. Unfortunately, I can't provide a hotfix right now.
My apologies that this piece failed for you.

OK, nice to hear. At least you know what the problem is. No rush, but do you have an estimated guess when you will have the time?

I would put next week. That is, the fix will be available in repository next week (at least, you can compile it yourself), but in PS gallery will appear in next PSPKI release.

I fixed the issue in the ASN.1 parser library: https://github.com/Crypt32/Asn1DerParser.NET/commit/644db6d2c4e12faeb43de86460de87513f6901df
now I can decode OCSP response properly:

For testing purposes I'm attaching compiled fixed ASN parser:
SysadminsLV.Asn1Parser.zip
Please, let me know if it works for you.

Nice. Its working again. Thank you very much.


[Bug]

/Hello,
I update for the last version and after use update .sh I get:
"
ERROR: relation "admin_settings" does not exist
"
tbmq logs:
`at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137)

... 62 common frames omitted

Caused by: org.postgresql.util.PSQLException: ERROR: relation "admin_settings" does not exist
Position: 152
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2676)

at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2366)

at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:356)

at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:496)

at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:413)

at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:190)

at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:134)

at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)

at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)

at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:57)

... 99 common frames omitted

Starting TBMQ ...
===================================================
:: ThingsBoard MQTT Broker ::       (v1.1.0)
===================================================`
postgres log:
`dminsetti0_.key as key4_0_ from admin_settings adminsetti0_ where adminsetti0_.key=$1
2023-09-18 17:17:08.180 UTC [597] ERROR:  relation "admin_settings" does not exist at character 152
2023-09-18 17:17:08.180 UTC [597] STATEMENT:  select adminsetti0_.id as id1_0_, adminsetti0_.created_time as created_2_0_, adminsetti0_.json_value as json_val3_0_, adminsetti0_.key as key4_0_ from admin_settings adminsetti0_ where adminsetti0_.key=$1
2023-09-18 17:17:20.437 UTC [613] ERROR:  relation "admin_settings" does not exist at character 152
2023-09-18 17:17:20.437 UTC [613] STATEMENT:  select adminsetti0_.id as id1_0_, adminsetti0_.created_time as created_2_0_, adminsetti0_.json_value as json_val3_0_, adminsetti0_.key as key4_0_ from admin_settings adminsetti0_ where adminsetti0_.key=$1
2023-09-18 17:17:30.001 UTC [630] ERROR:  relation "admin_settings" does not exist at character 152
2023-09-18 17:17:30.001 UTC [630] STATEMENT:  select adminsetti0_.id as id1_0_, adminsetti0_.created_time as created_2_0_, adminsetti0_.json_value as json_val3_0_, adminsetti0_.key as key4_0_ from admin_settings adminsetti0_ where adminsetti0_.key=$1
2023-09-18 17:17:40.763 UTC [646] ERROR:  relation "admin_settings" does not exist at character 152
2023-09-18 17:17:40.763 UTC [646] STATEMENT:  select adminsetti0_.id as id1_0_, adminsetti0_.created_time as created_2_0_, adminsetti0_.json_value as json_val3_0_, adminsetti0_.key as key4_0_ from admin_settings adminsetti0_ where adminsetti0_.key=$1
2023-09-18 17:17:51.088 UTC [663] ERROR:  relation "admin_settings" does not exist at character 152
2023-09-18 17:17:51.088 UTC [663] STATEMENT:  select adminsetti0_.id as id1_0_, adminsetti0_.created_time as created_2_0_, adminsetti0_.json_value as json_val3_0_, adminsetti0_.key as key4_0_ from admin_settings adminsetti0_ where adminsetti0_.key=$1
2023-09-18 17:18:01.341 UTC [680] ERROR:  relation "admin_settings" does not exist at character 152
2023-09-18 17:18:01.341 UTC [680] STATEMENT:  select adminsetti0_.id as id1_0_, adminsetti0_.created_time as created_2_0_, adminsetti0_.json_value as json_val3_0_, adminsetti0_.key as key4_0_ from admin_settings adminsetti0_ where adminsetti0_.key=$1
2023-09-18 17:18:11.347 UTC [696] ERROR:  relation "admin_settings" does not exist at character 152
2023-09-18 17:18:11.347 UTC [696] STATEMENT:  select adminsetti0_.id as id1_0_, adminsetti0_.created_time as created_2_0_, adminsetti0_.json_value as json_val3_0_, adminsetti0_.key as key4_0_ from admin_settings adminsetti0_ where adminsetti0_.key=$1
2023-09-18 17:18:22.131 UTC [713] ERROR:  relation "admin_settings" does not exist at character 152
2023-09-18 17:18:22.131 UTC [713] STATEMENT:  select adminsetti0_.id as id1_0_, adminsetti0_.created_time as created_2_0_, adminsetti0_.json_value as json_val3_0_, adminsetti0_.key as key4_0_ from admin_settings adminsetti0_ where adminsetti0_.key=$1`

It works after remove manually all content from :
sudo rm -R ~/.tb-mqtt-broker-data

What OS are you using? - ubuntu 20.04
Do you use Docker Engine or Docker Desktop? docker engine - portainer logs
Was the version of TBMQ from which you upgraded v1.0.1? 1.0.1 to 1.1.0
Was the result of ./tbmq-upgrade.sh script successful? yes, but never work anymore if I don't remove the "old" folder

@jot4p,
On an Ubuntu 20.04 machine, I have done the following:
Downloaded the docker-compose.yml file with the following tbmq version:
image: "thingsboard/tbmq:1.0.1"
Then installed this version using:
wget https://raw.githubusercontent.com/thingsboard/tbmq/main/msa/tbmq/configs/tbmq-install-and-run.sh && sudo chmod +x tbmq-install-and-run.sh && ./tbmq-install-and-run.sh

And finally upgraded it to 1.1.0 using:
wget https://raw.githubusercontent.com/thingsboard/tbmq/main/msa/tbmq/configs/tbmq-upgrade.sh && sudo chmod +x tbmq-upgrade.sh && ./tbmq-upgrade.sh

All went successfully, so it is not possible for me to reproduce the issue for now. Please correct me or the steps I have done if they differ in your case.

@jot4p are you able to provide any feedback regarding the above info?

Closing due to inactivity. Please, reopen with additional comments otherwise.


Build fails on Python 3.10 due to the collections module

gcloud fails to load on Python 3.10:
ERROR: gcloud failed to load: module 'collections' has no attribute 'Mapping'

Also, lint and unit tests fail to run on Python 3.10, example run here:
https://github.com/google/slo-generator/runs/4247056898?check_suite_focus=true

Looks like it is a known issue which got fixed a couple of days ago ((Mar 17, 2022) in release 377.0.0: https://issuetracker.google.com/issues/202172882?pli=1
Let me install Python 3.10.2 and confirm make lint and make unit complete successfully.

make lint still fails but I am not surprised as the issue with gcloud is completely independent:
(venv10.3) $ make lint
flake8 --ignore=E302,E203,E261 "slo_generator"/ --max-line-length=80
flake8 --ignore=E302,E203,E261,E402 tests/ --max-line-length=80
find ./"slo_generator" ./tests -name \*.py | xargs pylint --rcfile .pylintrc --ignore-patterns=test_.*?py
************* Module slo_generator.utils
slo_generator/utils.py:422:24: E1101: Module 'collections' has no 'Mapping' member (no-member)

------------------------------------------------------------------
Your code has been rated at 9.97/10 (previous run: 9.97/10, +0.00)

make: *** [Makefile:81: pylint] Error 123

make unit fails too. Again, this is expected:
(venv10.3) $ make unit
[...]
  File "/home/laurent/workspace/github/google/slo-generator/venv3.10/lib/python3.10/site-packages/nose/suite.py", line 106, in _set_tests
    if isinstance(tests, collections.Callable) and not is_suite:
AttributeError: module 'collections' has no attribute 'Callable'
make: *** [Makefile:69: unit] Error 1

Upon investigation, there is an easy fix (replace import collections with import collections.abc and collections.Mapping with collections.abc.Mapping at call sites) but it requires updating the nose library itself. Unfortunately, nose is no longer being maintained and the latest 1.3.7 version is explicitly not compatible with Python >= 3.10:

Nose is incompatible with Python >= 3.10 #95
nose is unmaintained and will stop working under future versions of Python #942

And the official website starts with:

Nose has been in maintenance mode for the past several years and will likely cease without a new person/team to take over maintainership. New projects should consider using Nose2, py.test, or just plain unittest/unittest2.

@ocervell WDYT? Is this a good opportunity to move to py.test?

@lvaylet  Sounds good to me. Had no idea nose was going to be discontinued...

Fixed by:

https://github.com/google/slo-generator/pull/256
https://github.com/google/slo-generator/pull/254


Support pre-shared key

Wireguard tunnels may be encrypted with an optional, pre-shared key. It adds an additional layer of symmetric-key  cryptography  for post-quantum resistance. This PR adds --endpoint-preshared-key /ONETUN_ENDPOINT_PRESHARED_KEY to specify such a key.

Thank you for the PR! I will review this shortly

Thank you for the quick review!


ORM Many-Many Duplicates

Something along these lines causes duplicate entries:
Model_One and Model_Two have a many many relation successfully set up.
$obj1 = Model_Onet::find($one_ID);
$obj2 = Model_Twot::find($obj1_ID);
$obj1->twos[] = $obj;
$obj1->save();
//so far so good.
$obj3 = Model_Two::find($obj3_ID);
$obj1->twos[] = $obj3;
$obj1->save();
//here, in the many many relation table, we find that $obj2 was saved again, along with $obj3. (twice $obj2, once $obj3)

//if we continue...
$obj4 = Model_Two::find($obj4_ID);
$obj1->twos[] = $obj4;
$obj1->save();
//now in the many many relation table, $obj2 is repeated 3 times, $obj3 twice, and $obj4 once.

doesn't the orm check whether the object was already saved?

Thank you for the bug report but this is the wrong repo. Please reopen you issue at https://github.com/fuel/orm


Update README.md

Bump version number to 1.2

This not only looks good to but inspiring as well :smile:

@nethip I think there was a misunderstanding -- in the thread above we were talking about using a different wording from what's in this PR, so we don't have to update the README every release.  So we wanted to wait to merge this until the wording was updated...  @coliff can you put up a new PR with the new wording?

@peterflynn Sure - are we going with:
'Brackets may have reached version 1, but we're not stopping there.'
Please confirm and I'll open a PR.

@peterflynn @coliff My Bad I don't know why I was under the impression that the suggested wording was already in the PR. I will review the new PR going to be uploaded by @coliff

@coliff

are we going with: 'Brackets may have reached version 1, but we're not stopping there.' ?
Please confirm and I'll open a PR

Yes


Fix start useRequest import

Fixed the import statement directed at the jsx file:
import { useRequest } from "solid-start/server/ServerContext.jsx";
Resolution:
import { useRequest } from "solid-start/server";

As I remember .jsx was because of NodeNext node resolution.
SolidStart doesn't include extensions when reexporting in server.ts, which breaks the resolution.
I would like to "fix" the import as well, but solid-start needs fixing first probably.
You can see it breaking the build.
Also we should probably move to @solidjs/start package.

Also we should probably move to @solidjs/start package.
Then let's get this merged first https://github.com/solidjs-community/solid-primitives/pull/585 and then I will do a followup pr that fixes the import.

Fixed in https://github.com/solidjs-community/solid-primitives/pull/660


BarrierClosure tests are not executed and BarrierClosureTest.MultipleRuns crashes

New unit tests for base::BarrierClosure helper was not added to CMakeLists script which means that they are not built and executed.

The intention was to do:

Expect N runs
Run N-1 times
Verify that the original callback hasn't been executed yet
Run Nth time
Verify that the original callback has been executed now

Unfortunately, the loop executes callback N times instead of N-1 times, which triggers CHECK in the last step (now N+1th instead of Nth).


Fix the :controller attribute in the quadicon url

In the Services List view, the quadicon in case of a grid or a tile setting, contains a wrong url with the controller value set to the the vm controller name. The vm controller name seems to match the type of vm that's contained in the service.
https://bugzilla.redhat.com/show_bug.cgi?id=1394503
Before:

After:

@miq-bot add_label ui,blocker,bug

@hayesr Does the default_url_options buy us anything here?
It's giving us a wrong controller name. Can we skip calling it altogether?

@AparnaKarve A recurring problem in quadicons is missing url parts (:controller, :action, etc.) We run into problems like this one and all kinds of testing headaches.
Preferably we'd be specific about which controller we want to link to. When we aren't, Rails calls default_url_options via url_for anyway to fill in the gaps. I probably used it here to start the attrs hash and to call attention to that being where the missing info was going to come from anyway. Better to do what you did, though it's weird that it was giving a different controller value.
I'm in the process of refactoring all of this so that call will go away :)

This is related to https://github.com/ManageIQ/manageiq/pull/12638 which did the same thing for vm_or_template. Just adding here so that we can use the same approach in the future.. :)
(And re-kicking the tests..)

Just adding here so that we can get some common approach in the future.. :)

@himdel The approach that I have taken addresses the issue at source, which means we won't see any quadicon url issues in the future (I hope :))
In that case, we don't need the fix in #12638 anymore after this gets merged.

Great news, let's hope so, thanks @AparnaKarve :). Either way, something to look for when re-refactoring :)

Euwe Backport details:
$ git log -1
commit 33b6bc710882895c66942f1141e0267b24f42355
Author: Dan Clarizio<EMAIL_ADDRESS>Date:   Wed Nov 16 08:36:49 2016 -0800

    Merge pull request #12662 from AparnaKarve/bz1394503_fix_quadicon_link

    Fix the :controller attribute in the quadicon url
    (cherry picked from commit 72427b137597b71456a5831a0cdc2f698b4f657f)

    https://bugzilla.redhat.com/show_bug.cgi?id=1394503


Reordering Custom Doc Type Editors on a page not allowed

Unable to order custom doc type editors (Custom overridden LeBlender controllers). Information and details on recreating the issue can be found here:
[https://our.umbraco.org/projects/backoffice-extensions/doc-type-grid-editor/doc-type-grid-editor-feedback/75665-reordering-doc-type-editors-on-a-page-not-allowed]

For some reason the grid layout data type did not want to save mu option to include all editors. Had to create new one and transfer existing widgets / editors over.

Hi @nkleroux - I haven't experienced this myself - but I never used the LeBlender, so not sure what complications that may bring to it?  Glad to hear that you've been able to resolve it. 😃


CRAN section

Not sure if this in the scope of the package but we can add a section if a package is on CRAN and since how long. We might also add the package authors (see also #20 for this).  Since if a package is on CRAN we know that it has passed the CRAN review and gives some trust. Some packages that we can use https://www.r-bloggers.com/2019/07/pkginfo-tools-for-retrieving-r-package-information/

Maybe this could be done in part by having the cran badge (that we use on readme) at the top of the report?
Then for cran stats, adding number of downloads and number of reverse dependencies would be a useful metric I think

I left some comments in the PR.

As its been a month, I've closed the PR for now, I'll do the same with this issue


tf-idf.Rmd

The difference between the # terms in the HashingTf feature and the # distinct terms in the sentence is due to hashing collision, not the exclusion of the last term.

hi @joehale-mfj , thanks for submitting a PR! Could you edit this file (https://github.com/MingChen0919/learning-apache-spark/blob/master/TF-IDF.ipynb) and re-submit the PR? the R Markdown files are not maintained anymore.

done


Cannot lint tag commits: unknown revision or path not in the working tree

Whenever TravisCI builds a tag, conventional-changelog-lint fails. It is working fine when TravisCI builds a pull request (in a branch).
Any ideas? Is my lint command wrong? The command is:
conventional-changelog-lint --from master

Node version: v4.4.7
npm version: 3.10.3
conventional-changelog-lint version: 1.0.0

Example of a TravisCI failure: https://travis-ci.org/mozilla/sign-addon/jobs/143130310
Complete error:
/home/travis/build/mozilla/sign-addon/node_modules/conventional-changelog-lint/distribution/cli.js:163
	throw error;
	^
Error: fatal: ambiguous argument 'master..HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
    at DestroyableTransform._transform (/home/travis/build/mozilla/sign-addon/node_modules/conventional-changelog-lint/node_modules/git-raw-commits/index.js:54:30)
    at DestroyableTransform.Transform._read (/home/travis/build/mozilla/sign-addon/node_modules/through2/node_modules/readable-stream/lib/_stream_transform.js:159:10)
    at DestroyableTransform.Transform._write (/home/travis/build/mozilla/sign-addon/node_modules/through2/node_modules/readable-stream/lib/_stream_transform.js:147:83)
    at doWrite (/home/travis/build/mozilla/sign-addon/node_modules/through2/node_modules/readable-stream/lib/_stream_writable.js:313:64)
    at writeOrBuffer (/home/travis/build/mozilla/sign-addon/node_modules/through2/node_modules/readable-stream/lib/_stream_writable.js:302:5)
    at DestroyableTransform.Writable.write (/home/travis/build/mozilla/sign-addon/node_modules/through2/node_modules/readable-stream/lib/_stream_writable.js:241:11)
    at Socket.ondata (_stream_readable.js:542:20)
    at emitOne (events.js:82:20)
    at Socket.emit (events.js:169:7)
    at readableAddChunk (_stream_readable.js:153:18)

If I check out a tag in my local repo I cannot reproduce the error. I'm not sure exactly how TravisCI is checking out the tag.

Oh, huh, I guess this is happening on some branch commits too. Example: https://travis-ci.org/mozilla/web-ext/jobs/143135532 Most pull requests seem to be fine though.

I see what's happening. TravisCI does a shallow clone, something like this (depending on pull request and repo):
git clone --depth=50 --branch=greenkeeper-sign-addon-0.1.0 https://github.com/mozilla/web-ext.git mozilla-web-ext

This does not fetch the master branch. I fixed it by running this before installation (in my .travis.yml):
git remote set-branches origin master && git fetch

@kumar303 Thanks for reporting! Glad you solved the problem 🚀  – I created a followup issue at #7  anyway.


Bump react-native-gesture-handler to latest version

The motivation behind this PR is that ViewPagerAndroid has been deprecated and moved to react-native-community.  That change has been made in the last version of  react-native-gesture-handler
So we can avoid the warnings like:
  console.error node_modules/fbjs/lib/warning.js:30
      Warning: ViewPagerAndroid has been extracted from react-native core and will be removed in a future release. It can now be installed and imported from '@react-native-community/viewpager' instead of 'react-native'. See https://github.com/react-native-community/react-native-viewpager

thx

I released this in 3.5.0-alpha.0


Cursor position blocks brief text in olPopup div

Issue: The cursor can sometimes block a portion of the brief text in the olPopup (Note: This occurs more often when the cursor is positions left of centre, and when text is LtoR in orientation).
Suggestion: It may be useful to alter the position of the olPopup to be further away from the cursor position.

This has been resolved with style updates in the mdc_integration branch.


:sparkles: move the space client lib into the space-framework

Create a space client lib in the space-framework
Remove the KS references in the client lib  (TODO: need to find a different approach for MS clientset)
For now leave the replacement statement in go.mod although space-framework is not used.

Summary
Related issue(s)
Fixes #

VSCode gives me these warnings about the Space go.mod:
[{
	"resource": "/Users/mspreitz/go/src/github.com/kubestellar/kubestellar/space-framework/go.mod",
	"owner": "_generated_diagnostic_collection_name_#0",
	"severity": 4,
	"message": "k8s.io/code-generator is not used in this module",
	"source": "go mod tidy",
	"startLineNumber": 14,
	"startColumn": 2,
	"endLineNumber": 14,
	"endColumn": 31
},{
	"resource": "/Users/mspreitz/go/src/github.com/kubestellar/kubestellar/space-framework/go.mod",
	"owner": "_generated_diagnostic_collection_name_#0",
	"severity": 4,
	"message": "golang.org/x/mod is not used in this module",
	"source": "go mod tidy",
	"startLineNumber": 31,
	"startColumn": 2,
	"endLineNumber": 31,
	"endColumn": 59
},{
	"resource": "/Users/mspreitz/go/src/github.com/kubestellar/kubestellar/space-framework/go.mod",
	"owner": "_generated_diagnostic_collection_name_#0",
	"severity": 4,
	"message": "golang.org/x/tools is not used in this module",
	"source": "go mod tidy",
	"startLineNumber": 32,
	"startColumn": 2,
	"endLineNumber": 32,
	"endColumn": 28
},{
	"resource": "/Users/mspreitz/go/src/github.com/kubestellar/kubestellar/space-framework/go.mod",
	"owner": "_generated_diagnostic_collection_name_#0",
	"severity": 4,
	"message": "k8s.io/gengo is not used in this module",
	"source": "go mod tidy",
	"startLineNumber": 34,
	"startColumn": 2,
	"endLineNumber": 34,
	"endColumn": 49
}]

(venv) mspreitz@mjs13 kubestellar % pushd space-framework
~/go/src/github.com/kubestellar/kubestellar/space-framework ~/go/src/github.com/kubestellar/kubestellar ~/go/src/github.com/kubestellar/kubestellar/pkg/placement

(venv) mspreitz@mjs13 space-framework % go mod tidy

(venv) mspreitz@mjs13 space-framework % git status
On branch spaceclient
Your branch is up to date with 'ezra/spaceclient'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   go.mod
	modified:   go.sum

VSCode gives me these warnings about the Space go.mod:

I cleaned up the go.mod. Hopefully it fixed the warnings

squashed commits

/hold  addressing comments

/unhold


Some directories do not delete

Going to restate a little bite from previsus now to be closed issue #15.
I significantly altered the provided example to do stress testing. I made it run a loop from .begin to .end to find any leaks and determine robustness. Once an application level error was addressed. No leaks.
To the problem at hand...
I added code to delete all created files and directories. I found that some directories resist .rmdir. I tried several methods of exclusion to isolate the issue but have not found it.
Specifically, all files delete properly. some directories do not seem to respond to rmdir. These do not remove:
/this
/this/folder
/this/folder/does
/this/folder/does/really
My suspicion is a combination from PSRamFS_test.ino lines:
492:   writeFile(PSRamFS, "/this/folder/does/not/exist/yet/goodbye.txt", "you say hello, I say goodbye"); // should succeed and create all subfolders
and
505:  PSRamFS.rename("/this/folder/does/not", "/this/folder/does/really" );
I am writing a from scratch "sketch" to see if I can isolate the source in a smaller sketch or determine that it is application level with example.

hi,
the description is consistent with the state of the code, folder implementation in pfs.c is incomplete
PSRamFS codebase initially reused directory support model from SPIFFS (see struct _pfs_dir_t) but openDir behaved strangely and the unit tests wouldn't pass; so a couple of changes were applied to add recursion but since SPIFFS directory model is flawed (which I didn't know at the time of writing the code) it only pushed the problems one level deeper.
then later I discovered that SPIFFS directory support is incomplete 🤦
as lame as it may sound, PSRamFS directory support is broken because the model it relies upon is flawed, and it can only be fixed by using a better model
obviously PSRamFS should use LittleFS directory model instead (i.e. struct lfs_mlist and lfs_info instead of _pfs_dir_t/_pfs_file_t linked lists), but this requires a massive refactoring of the vfs layer (pfs.c/pfs.h) 😅

Thank you for creating and maintaining it. I am willing to help if you want.
The library is truly great to have, other options do not suit my needs.
The explanation works for me.
Converting to LittleFS model = large effort. Not sure if justified as my use case is not as complicated as the provided example even though I and deep diving into the library.
We can close this one.


Link to Cantera documentation in users guide results in 404 error.

Minor issue note:
The link to Cantera documentation (http://cantera.github.io/docs/sphinx/html/index.html) provided at the top of section 5 (on page 12) of the user's guide Microsoft Word document does not work. The link results in a 404 error when followed.

Thanks!  This has been updated within the documentation.


Update dynamic host prop even without dynamic port prop

Fixes #323
Signed-off-by: Caspar Krieger <email goes here>
Replaces #324 .

Thanks a lot, looks good to me.


Ignores displayname files

Hello,
Summary
According to the documentation, todoman searches for a file named displayname in every directory name, and uses its raw contents as the list name. If not found, todoman uses the folder name as list name.
Version
In my installation (3.5.0 installed from packages on Debian 10) this behavior is not happening.
Configuration
I have two folders (i.e. two task lists), and on each of them there is a file named displayname.
The files can be read and the content is a string containing the list name.
~$ cat /home/[username]/.local/share/vdirsyncer/storage/cd77043e-[redacted]/displayname
Tasks
~$ cat /home/cfenollosa/.local/share/vdirsyncer/storage/A0EE9E61-[redacted]/displayname
Futuro

This is my config file:
~$ cat .config/todoman/todoman.conf
[main]
path = ~/.local/share/vdirsyncer/storage/*
default_list = Tasks
default_due = 0
humanize = True

As you can see, my default list is "Tasks", i.e., folder cd77043e-[redacted]
I have also confirmed that my version contains the code to read the displayname:
~$ grep -R "displayname" /usr/lib/python3/dist-packages/todoman/
/usr/lib/python3/dist-packages/todoman/model.py:            with open(os.path.join(path, 'displayname')) as f:
Binary file /usr/lib/python3/dist-packages/todoman/__pycache__/model.cpython-37.pyc matches

Just in case, strace confirms that the displayname files are read
~$ strace todoman 2>&1 | grep displayname
openat(AT_FDCWD, "/home/[username]/.local/share/vdirsyncer/storage/cd77043e-[redacted]/displayname", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/home/[username]/.local/share/vdirsyncer/storage/A0EE9E61-[redacted]/displayname", O_RDONLY|O_CLOEXEC) = 4

Behavior
todoman is not using the content of these files when displaying tasks
~$ todoman
42  [ ]                   Task one @cd77043e-[redacted]
10  [ ]                   Task two @cd77043e-[redacted]

(more output follows, snipped)

todoman is not using the displayname when adding tasks, either.
Note that I have configured a default list by using their displayname.
~$ todoman new test
Usage: todoman new [OPTIONS] [SUMMARY]...

Error: Invalid value for "--list" / "-l": Tasks. Available lists are: cd77043e-[redacted], A0EE9E61-[redacted]

When using the folder name as list name, todoman works as expected
~$ todoman new test --list cd77043e-[redacted]
47  [ ]      test @cd77043e-[redacted]

Conclusion
Is there something wrong in my setup or is this maybe a bug?
Thanks!

Okay, I did some more research and found the culprit and the solution.
After reading todoman source code, I found a suspicious keyword: cache. Now it all makes sense.
Leaving a comment here so other people can find it if they ever encounter the problem.
Problem
The initial run of todoman was executed when the task folders did NOT have a displayname file. This is because vdirsyncer metasync did not run and therefore didn't create this file.
The displayname was added later, manually.
Since todoman uses a cache, that cache is initialized but never updated whenever the displayname or color files appear.
Solution
Delete the cache and run todoman again
~$ mv ~/.cache/todoman/cache.sqlite3{,.bak}
~$ todoman

Further work
I suggest that todoman updates their cache if it finds the displayname or color files anew, or whenever the content of these files changes.
I'm leaving the issue opened so that an admin can read this last comment, and decide whether it would be a good idea to update the caches.
Admins: Feel free to close this issue. Thanks for reading.

You version of todoman is from two years ago.This bug was fixed in January 2018.


Form claims should Time Out

As a SuperUser/User, I want to know that a form I need to update is free, and that it is safe to input the information I need. I also want to know that if I forget to "unclaim" a form, that my claim will time out so that others can use the form.

12 hours after last save - kick users out of claimed form


Resistor Band: Fill in dummy functions

[x] Tidy up the code
[x] Make sure the resistor band works when you press on it


Andrey #353. Ignore twitter errors

So if it is an error from twitter's widgets.js - we will ignore it.
If that is a different error - we will record a video for next 60 seconds to check what is going on.
Later we can just make a simple list of errors (like all errors initiated from a widgets.js) which we are not going to track.

Netlify? Waiting for it!

Here it is - https://5aa16888df99536ce361d824--landscape.netlify.com/

Deploy preview for landscape ready!
Built with commit b62f7b768abfe6b5a2543908db55712ac6d2f731
https://deploy-preview-354--landscape.netlify.com


Support for an initial state

I implemented the possibility to pass an initial state to the MuiDataTable plugin. This initial state, does what it says, sets the already existed state. I had to re-arrange some of the logic to accept an initial state, since it wasn't build for that.

Moved page and rowsPerPage from options to the new initialState prop
Fallback support for page and rowsPerPage under options prop
Implemented searchText, filters, sort and columnOptions (for visibility)
Supports serverside: true
Adjusted readme
Added example

Coverage increased (+0.3%) to 72.65% when pulling 557fb387f95c4e2aa9b283d74e76740aee57fe87 on nrdev88:master into 755809dacb49011098a419fb05e3bd957503f728 on gregnb:master.

Thanks for this contribution. I am trying it out.
Given that you moved rowsPerPage and page to the new prop would it make sense to move count as well?

Filters looks strictly equivalent to filterList. I think the name change is a bit confusing.

It would be nice to have initialState members optional. It currently fails if you don't supply a sort member.

Thanks for this contribution. I am trying it out.
Given that you moved rowsPerPage and page to the new prop would it make sense to move count as well?

I did think about it, but somehow it didn't make sense for me. I just don't know why anymore, can't find any good reasons not too.

Filters looks strictly equivalent to filterList. I think the name change is a bit confusing.

I find filterList confusing. But I can change that. You're right about consistency.

It would be nice to have initialState members optional. It currently fails if you don't supply a sort member.
Edit1: I see that you are merging defaults, not sure what is going on. Hold on..
Edit2: OK, I had set sort = null instead of sort.column and sort.direction.

True point. Should fix that :-)

Just to be clear, I am not a contributor to this project.. just some random dude and this PR is just exactly what I needed this morning. I am already using it, it's awesome. Thanks!

hey @nrdev88 thanks for the PR. This is quite a large change. I'll need some time to think about this

I would love to see this merged :+1:

Is there any update on this PR?  Would really love to see this added, but if there's any issue holding it up, could they be shared so we potentially address them and move this along? :)

@gregnb Do you need any help merging this? This could be very helpful to hydrate the table, with previous saved state.

@gregnb I'm also very interested in this change. Would be great to see it merged.

So my questions are why would we need an initialState? Is there something you can not already do by the available options? I don't want the package to get to the point where we have multiple options/props that do the same thing

We need that so when the user hit back he/she can resume at the same page,
with the same filters/search/etc. We need a way to rehydrate the table.
On Sat., Apr. 13, 2019, 12:03 p.m. gregn<EMAIL_ADDRESS>wrote:

So my questions are why would we need an initialState? Is there something
you can not already do by the available options? I don't want the package
to get to the point where we have multiple options/props that do the same
thing
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/gregnb/mui-datatables/pull/441#issuecomment-482827740,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAr2MJqrtgLENeowmZLh8vu5IhZKi6puks5vgf_LgaJpZM4bPtDK
.

@jpambrun To be more specific what options are not currently available as props that are you stopping from rehydrating the table?

Currently you can’t set options for selected filter values, current sorted column, etc.

Anyone interested in this thread can we please list out the options they are missing when it comes to rehydrating the state. When we compile a list I'll look into it and see if someone isn't aware of an option and what is really missing that needs to be added.
@nrdev88 Selected filter values: https://github.com/gregnb/mui-datatables/blob/master/examples/column-filters/index.js

Current filter + sort column+order is needed for the exact same purpose. When table is unmounted I am going to save current table state, then when user hits the back button, he can pick where he left off.

@gregnb We have the same requirements as @corroleaus and have implemented what we need in a fork. We didn't want to duplicate functionality from this PR so haven't submitted our own PR yet but you can take a look: https://github.com/continuum-media/mui-datatables

@gregnb I also have the use case where I would like to rehydrate the table with previous user settings. For my specific requirements I needed everything addressed in this PR. I am currently just using
To list it out:

page (current page)
rowsPerPage
searchText
filters (for all types)
column sort (ASC or DESC, which column)
column order
column visibility

All effort on this is much appreciated.

Any idea for the status of this PR?

@paulmichaelxd @gregnb Correct me if I'm wrong.

page (current page)
rowsPerPage

can be hydrated using page and rowsPerPage of options prop

filters (for all types)
column visibility

can be hydrated using filterList and display column level options.

column order

can be achieved by ordering the columns as required in columns prop

column sort (ASC or DESC, which column)
searchText

These are the things which are actually missing.

@karthick-vinod Thank you for the summary!
We also have the column option sortDirection which sets the default sort order. I think this could be used for state hydration of the column order, right? If so, it seems that we only have one option needed, which would be searchText.
@nrdev88 What I would like to see, perhaps in a new PR (since this one has fallen out of date and has a lot of prettier changes), is an option added for searchText and maybe something for column order if sortDirection isn't enough for some reason. I think we should also leave the prettier changes to perhaps a separate PR, as it adds quite a lot of noise to the diffs and makes it difficult to review the essential features here. I would be happy to accept such a PR. What do you think?

@gabrielliwerant You are correct.

searchText: Please find my PR for searchText option here: https://github.com/gregnb/mui-datatables/pull/603

columnSort: As sortDirection is a column level option, assigning sortDirection to multiple columns actually sorts only the last column though Arrow Icon could be seen next to multiple columns according to sortDirection option specified. This might be a bug.

@karthick-vinod Yep, saw your PR just after I commented! I left a review, but looks great!

Ok, so the sortDirection behavior of only affecting the final column I believe is intentional, as multiple columns cannot be sorted simultaneously, as this require reordering individual cell data among rows, thus scrambling the table. However, it is unclear what is happening from a UI perspective, as it does show both arrows.
What should probably occur is that only the final arrow should show, and there should be a console warning that multiple columns have been set to be sorted.
I opened an issue around this here https://github.com/gregnb/mui-datatables/issues/605.

This should be resolved via https://github.com/gregnb/mui-datatables/pull/603


[feature request] Ability to import external provisioners in init

It would be very helpful to be able to import existing shared templates coming from public/private Git repos and  public/private OCI registries.
Today, I need to manually copy any custom provisioners from Workload/Repo to Workload/Repo, not very convenient. And could lead to outdated version, inconsistency between workloads/projects, etc.
An idea could be to have a new --template or --import parameter in the score-compose init command?
This will add more provisioners in addition to the default ones.
Another idea, could also be to have a public repo in the score-spec Org where the community could share public shared provisioners, examples:

service
redis-dapr-state-store

@mathieu-benoit why not use curl or cp to pull in the provisioners?
In terms of the UNIX philosophy I'd prefer that score implementations didn't get involved in working out where to get custom provisioners from or how to authenticate against remote repositories. I'd prefer folks used one of the following approaches:

Check the custom provisioners into the repo in .score-compose/ and only add the .score-compose/{state.yaml,mounts,99-default.provisioners.yaml} to the gitignore.
curl / rsync a remote file or files as a second step in the developer instructions or Makefile

It gets complicated if we offload that to score init, because then it needs to start understanding how to authenticate and secure a remote file download, which frequently gets complicated inside developer networks.

Yeah, that's for sure that curl, wget, cp, rsync or oras can be used. And offloading the auth part outside score-compose init is indeed preferable too: using existing tools outside score-compose, not adding unnecessary maintenance for this in score-compose, etc.
Maybe just examples or explicit documentation could be added in score.dev to illustrate this is enough.

I would rather see a solution inside score-compose. If you're an enterprise platform team trying to provide a seamless platform experience to developers, then the ability to roll out a standard local resource set, which can also be kept current from a central point, is very valuable.
Trying to get hundreds, possibly thousands, of developers to consistently use a script wrapper spun around the tooling is not going to work in most cases.

Now supported: https://github.com/score-spec/score-compose/releases/tag/0.17.0, closing.