Calibrating for Wii Remote with MotionPlus

I'm trying to get pitch and roll values that in some way correspond to the real pitch and roll of the device. However, whenever I do a motion, the values seem to drift back to roughly where they started even if the Wii Remote is now in a different orientation. This is the script that I'm currently using:
from datetime import datetime
from fusion import Fusion
import cwiid
import time

def timediff(time1, time2):
    return (time1-time2)

# This part is for connecting the Wii Remote.
wiimote = cwiid.Wiimote()
time.sleep(1)
wiimote.rpt_mode = cwiid.RPT_BTN | cwiid.RPT_ACC | cwiid.RPT_MOTIONPLUS | cwiid.RPT_IR
wiimote.enable(cwiid.FLAG_MOTIONPLUS)
time.sleep(1)

fuse = Fusion(timediff)

count = 0
while True:
    # This ordering of values seems to correspond most closely to the order yaw, pitch, and then roll. Though, the "yaw" doesn't give intuitive values.
    # Tilting the Wiimote upward in pitch, the ['acc'][1] gets lower. Tilting it down, the ['acc'][1] gets higher.
    # Rolling it counterclockwise, the ['acc'][0] gets lower. Rolling clockwise, the ['acc'][0] value gets higher.
    accel = (wiimote.state['acc'][2], wiimote.state['acc'][1], wiimote.state['acc'][0])

    angle_rates = wiimote.state['motionplus']['angle_rate']

    # This ordering of values seems to correspond most closely to the order yaw, pitch, and then roll.
    # By tilting the Wiimote upward in pitch, the angle_rates[0] gets momentarily lower. Downward pitch gets it to go higher.
    # By rolling the Wiimote counterclockwise, the angle_rates[1] gets momentarily higher. Rolling it clockwise, it gets lower.
    # By swaying the Wiimote toward the left in yaw, the angle_rates[2] gets momentarily higher. Swaying it toward the right, it gets lower.

    # Explanation for the dividing by low_speed times something bit:
    # When low_speed is 1, then 20 represents turning at about 1 degree per second. So divide by 20 to get the degrees per second.
    # At high speed (Low speed bit = 0) 20 represents turning at about 5 degree per second. So divide by 4 to get the degrees per second.
    # Source: http://arduino-projects4u.com/wii-motion-plus/

    # The -7945, -8115 and -7964 calibrate the values so that for me the gyro values will be close to zero when the Wii Remote is resting on the table.
    gyro = (
        (angle_rates[2]-7945)/(4+wiimote.state['motionplus']['low_speed'][2]*16),
        (angle_rates[0]-8115)/(4+wiimote.state['motionplus']['low_speed'][0]*16),
        (angle_rates[1]-7964)/(4+wiimote.state['motionplus']['low_speed'][1]*16),
    )
    fuse.update_nomag(accel, gyro, time.time())
    if count % 5 == 0:
        print(accel, gyro)
        print("Heading, Pitch, Roll: {:7.3f} {:7.3f} {:7.3f}".format(fuse.heading, fuse.pitch, fuse.roll))
    time.sleep(0.02)
    count += 1

What am I missing or doing wrong?

Maybe it's also not the algorithm I'm looking for anyway, since I'd want convergence to some pitch value in more like 0.1 seconds instead of the two that the README is saying. Raising GyroMeasError to 400 also seems to start giving values that are all over the place. But it might be good practice to see how it could be done with this library.

Maybe it's also not the algorithm I'm looking for anyway, since I'd want convergence to some pitch value in more like 0.1 seconds instead of the two that the README is saying. Raising GyroMeasError to 400 also seems to start giving values that are all over the place. But it might be good practice to see how it could be done with this library.

I've no experience with the Wii but it should work. However the behaviour you're getting is evidently wrong. You've probably already checked the following, but these are the thoughts that come to mind.
I assume your coordinate system conforms with this note. The Z axis accelerometer is easily identified by the 1G offset. You can check X and Y by rotating it as per that note and observing what happens to the offset. Gyro axes surely correspond unless the device is perverse; they can be checked similarly by orientating it on each orthogonal axis, running a loop that displays Gyro X, Y, Z repeatedly, and giving it a quick twist. Note that the gyro must produce degrees per second.
I haven't experimented with Beta very much: I've no idea whether 100ms convergence is feasible. I believe there are other more modern algorithms but I haven't studied them. I think Madgwick tends to prioritise accuracy over speed whereas most Wii applications require the opposite. Perhaps something can be done with Kalman filters or complementary filters?

I've no experience with the Wii but it should work. However the behaviour you're getting is evidently wrong. You've probably already checked the following, but these are the thoughts that come to mind.
I assume your coordinate system conforms with this note. The Z axis accelerometer is easily identified by the 1G offset. You can check X and Y by rotating it as per that note and observing what happens to the offset. Gyro axes surely correspond unless the device is perverse; they can be checked similarly by orientating it on each orthogonal axis, running a loop that displays Gyro X, Y, Z repeatedly, and giving it a quick twist. Note that the gyro must produce degrees per second.
I haven't experimented with Beta very much: I've no idea whether 100ms convergence is feasible. I believe there are other more modern algorithms but I haven't studied them. I think Madgwick tends to prioritise accuracy over speed whereas most Wii applications require the opposite. Perhaps something can be done with Kalman filters or complementary filters?

The comments in the code describe a lot of the "whys" of what's there. It's a bit difficult for me to understand the coordinate systems, but leaving the accelerometer and gyro values in their default order didn't seem to produce correct results so I tried ordering them according to the order "yaw, pitch, and then roll" mentioned under section 5.1 Heading Pitch and Roll. I think it's better explained in the comment about what twisting the device each way does to the values. I also don't know what the positive direction should be for each case. The direction for pitch's axis I'm pretty sure is correct, though.

Note that the gyro must produce degrees per second.

That's something I tried to take into account. The raw angle_rates reported by the gyro are first calibrated to be near zero at rest (I'm not sure if that was necessary but still). Then they are scaled by a value that to my understanding should mean they will be in degrees per second, but of course my reasoning about it could be wrong.
There's no calibration to the accelerometer values. I tried to have them be 0 at rest, but then there were some divisions by zero so I concluded it's probably not what's wrong.

Perhaps something can be done with Kalman filters or complementary filters?

I started reading up on Kalman filters after noticing the Madgwick algorithm may not be suitable.
Oh, and I forgot to add an example of the kind of output this code gives, in case it points out anything. In this case the device has been at rest on the table for enough time for the values to more or less converge:

(122, 123, 147) (0.1, -0.7, 0.7)
Heading, Pitch, Roll:   0.000 -33.079  40.817

The comments in the code describe a lot of the "whys" of what's there. It's a bit difficult for me to understand the coordinate systems, but leaving the accelerometer and gyro values in their default order didn't seem to produce correct results so I tried ordering them according to the order "yaw, pitch, and then roll" mentioned under section 5.1 Heading Pitch and Roll. I think it's better explained in the comment about what twisting the device each way does to the values. I also don't know what the positive direction should be for each case. The direction for pitch's axis I'm pretty sure is correct, though.

Note that the gyro must produce degrees per second.

That's something I tried to take into account. The raw angle_rates reported by the gyro are first calibrated to be near zero at rest (I'm not sure if that was necessary but still). Then they are scaled by a value that to my understanding should mean they will be in degrees per second, but of course my reasoning about it could be wrong.
There's no calibration to the accelerometer values. I tried to have them be 0 at rest, but then there were some divisions by zero so I concluded it's probably not what's wrong.

Perhaps something can be done with Kalman filters or complementary filters?

I started reading up on Kalman filters after noticing the Madgwick algorithm may not be suitable.
Oh, and I forgot to add an example of the kind of output this code gives, in case it points out anything. In this case the device has been at rest on the table for enough time for the values to more or less converge:

(122, 123, 147) (0.1, -0.7, 0.7)
Heading, Pitch, Roll:   0.000 -33.079  40.817

Hmm, this is interesting. I managed to get correct-looking numbers by adding a bias to the accelerometer values as well. Except, the value output as roll looks like that of pitch's and the value output for pitch looks like roll's. The while loop now looks like this:
while True:
    accel = (wiimote.state['acc'][0]-122, wiimote.state['acc'][1]-122, wiimote.state['acc'][2]-122)

    angle_rates = wiimote.state['motionplus']['angle_rate']
    gyro = (
        (angle_rates[0]-7945)/(4.0+wiimote.state['motionplus']['low_speed'][0]*16),
        (angle_rates[1]-8115)/(4.0+wiimote.state['motionplus']['low_speed'][1]*16),
        (angle_rates[2]-7964)/(4.0+wiimote.state['motionplus']['low_speed'][2]*16),
    )
    fuse.update_nomag(accel, gyro, time.time())
    if count % 5 == 0:
        print(accel, gyro)
        print("Heading, Pitch, Roll: {:7.3f} {:7.3f} {:7.3f}".format(fuse.heading, fuse.pitch, fuse.roll))
    time.sleep(0.02)
    count += 1

The output is now kind of like this:

(0, 1, 25) (7.4, -8.4, -0.6)
Heading, Pitch, Roll:   0.000   0.024   2.288

when level but

(1, -23, -1) (8.05, -7.75, -0.7)
Heading, Pitch, Roll:   0.000  -2.509 -91.775

when pointing upward and

Heading, Pitch, Roll:   0.000  -2.436  91.936
(1, 25, -1) (20.0, -4.0, 12.55)

when pointing downward. The pitch and roll are the wrong way around but otherwise it's looking good. It also reported facing backward as +-180, which the Kalman filter implementation I tried didn't.
However, with biases

accel = (wiimote.state['acc'][0]-122, wiimote.state['acc'][1]-123, wiimote.state['acc'][2]-122)

it crashes:
[mikko@localhost micropython-fusion]$ python3.7 fusiontest_wiimote.py
Traceback (most recent call last):
  File "fusiontest_wiimote.py", line 51, in <module>
    fuse.update_nomag(accel, gyro, time.time())
  File "/mnt/USB/opinnot/SpecuSim/micropython-fusion/fusion.py", line 91, in update_nomag
    norm = 1 / sqrt(s1 * s1 + s2 * s2 + s3 * s3 + s4 * s4)    # normalise step magnitude
ZeroDivisionError: float division by zero
code 1

Hmm, this is interesting. I managed to get correct-looking numbers by adding a bias to the accelerometer values as well. Except, the value output as roll looks like that of pitch's and the value output for pitch looks like roll's. The while loop now looks like this:
while True:
    accel = (wiimote.state['acc'][0]-122, wiimote.state['acc'][1]-122, wiimote.state['acc'][2]-122)

    angle_rates = wiimote.state['motionplus']['angle_rate']
    gyro = (
        (angle_rates[0]-7945)/(4.0+wiimote.state['motionplus']['low_speed'][0]*16),
        (angle_rates[1]-8115)/(4.0+wiimote.state['motionplus']['low_speed'][1]*16),
        (angle_rates[2]-7964)/(4.0+wiimote.state['motionplus']['low_speed'][2]*16),
    )
    fuse.update_nomag(accel, gyro, time.time())
    if count % 5 == 0:
        print(accel, gyro)
        print("Heading, Pitch, Roll: {:7.3f} {:7.3f} {:7.3f}".format(fuse.heading, fuse.pitch, fuse.roll))
    time.sleep(0.02)
    count += 1

The output is now kind of like this:

(0, 1, 25) (7.4, -8.4, -0.6)
Heading, Pitch, Roll:   0.000   0.024   2.288

when level but

(1, -23, -1) (8.05, -7.75, -0.7)
Heading, Pitch, Roll:   0.000  -2.509 -91.775

when pointing upward and

Heading, Pitch, Roll:   0.000  -2.436  91.936
(1, 25, -1) (20.0, -4.0, 12.55)

when pointing downward. The pitch and roll are the wrong way around but otherwise it's looking good. It also reported facing backward as +-180, which the Kalman filter implementation I tried didn't.
However, with biases

accel = (wiimote.state['acc'][0]-122, wiimote.state['acc'][1]-123, wiimote.state['acc'][2]-122)

it crashes:
[mikko@localhost micropython-fusion]$ python3.7 fusiontest_wiimote.py
Traceback (most recent call last):
  File "fusiontest_wiimote.py", line 51, in <module>
    fuse.update_nomag(accel, gyro, time.time())
  File "/mnt/USB/opinnot/SpecuSim/micropython-fusion/fusion.py", line 91, in update_nomag
    norm = 1 / sqrt(s1 * s1 + s2 * s2 + s3 * s3 + s4 * s4)    # normalise step magnitude
ZeroDivisionError: float division by zero
code 1

The units of the accelerometer don't matter as the algorithm normalises them, however typically I calibrate them to produce G as this is easy to understand. With the unit in its normal orientation the Z axis should then produce a value of 1, with the other axes nominally 0. By rotating the unit 90° about the X and Y axes in turn, you should see 1 or -1 on their accelerometer values.
As stated in the docs without a magnetometer heading values will be meaningless. It is physically impossible to deduce absolute heading from an accelerometer and a gyro.

The units of the accelerometer don't matter as the algorithm normalises them, however typically I calibrate them to produce G as this is easy to understand. With the unit in its normal orientation the Z axis should then produce a value of 1, with the other axes nominally 0. By rotating the unit 90° about the X and Y axes in turn, you should see 1 or -1 on their accelerometer values.
As stated in the docs without a magnetometer heading values will be meaningless. It is physically impossible to deduce absolute heading from an accelerometer and a gyro.

Anyway, I feel like I can work with this, and it's much faster convergence, now that it's been calibrated, than before. Fast enough that it's at least a candidate for my purposes. I'm closing the issue.
Yeah, the heading will require a different tactic. The 180 thing was when I was pitching the remote so much it was upside down facing backward. Another thing I have access to is a "sensor bar" producing two infrared lights, which will give at least a starting point for the heading. I'm hoping that I can deduce the heading roughly from the last known position, the estimated roll over time and the gyro's data over time. But that's another topic to research.

Anyway, I feel like I can work with this, and it's much faster convergence, now that it's been calibrated, than before. Fast enough that it's at least a candidate for my purposes. I'm closing the issue.
Yeah, the heading will require a different tactic. The 180 thing was when I was pitching the remote so much it was upside down facing backward. Another thing I have access to is a "sensor bar" producing two infrared lights, which will give at least a starting point for the heading. I'm hoping that I can deduce the heading roughly from the last known position, the estimated roll over time and the gyro's data over time. But that's another topic to research.


Review documentation for extensions

Current extensions documentation is somehow outdated and misses some relevant pieces of information ('FORCE INSTALL' for example, or setting alternatives S3 buckets).
Basically most information from README.md of https://github.com/duckdb/extension-template could probably have a place somewhere in the docs.

Also noting that https://duckdb.org/dev/building#extension-flags is now outdated, and should be eventually realigned.

Related to ##917


Chat bot example

Testing new chat bots should work again, can add to example at some point for XOXO

Chat bot is not a client feature. This should be implemented on the server side,


[PE-D][Tester E] Mismatch in group name in delete command

Group name and delete command do not match up. The command states CS2103T but the description says CS2013-T15

Labels: severity.Low type.DocumentationBug
original: whelan-low/ped#5

fixed here


SYCL. Add basic functional for QuantileHistMaker

Hi,
I continue adding of sycl support (https://github.com/dmlc/xgboost/pull/10138, https://github.com/dmlc/xgboost/pull/10119  etc).
By this PR I add the next part of the training-related code for the sycl devices. Here I add basic logic related to hist maker and the related tests. Now it is possible to create sycl hist maker with TreeUpdater::Create.
This PR itself doesn't add any new functional, but I plan to combine all related parts one by one to a launchable sycl-training.

Hi @trivialfis , could we merge it?


Fix unresolved external symbol for MSVS 2017 (#3762)

If char16_t is used as template argument, this leads to an unresolved symbol for MSVS 2017. Use int16_t instead of char16_t in this case.
Fixes #3762
Checklist:

[x] I have read the contribution guidelines.
[x] If this is more extensive than a small change to existing code, I
have previously submitted a Contributor License Agreement
(individual, and if there is any way my
employers might think my programming belongs to them, then also
corporate). This is just a small change
[x] I have updated the documentation, if applicable. Not applicable.
[x] I have ensured that the change is tested somewhere in the testsuite
(adding new test cases if necessary). Not applicable.
[x] My code follows the prevailing code style of this project.

LGTM, thanks for the fix!
Do you need a tagged release right away?

@lgritz If that is possible, that would be great! But if it's too much effort, I can also just include the patch in the Conan package


Value Error in --mode='train'

I was trying your code on A Mac OS machine and I got this error in python 2.7:
ValueError: Variable discriminator/conv1/weights/Adam/ already exists, disallowed. Did you mean to set reuse=True in VarScope? Originally defined at:
File "/Users/arjun/tensorflow/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/learning.py", line 467, in create_train_op
grad_updates = optimizer.apply_gradients(grads, global_step=global_step)
File "/Users/arjun/dtn-tensorflow/model.py", line 129, in build_model
self.d_train_op_src = slim.learning.create_train_op(self.d_loss_src, self.d_optimizer_src, variables_to_train=d_vars)
File "/Users/arjun/dtn-tensorflow/solver.py", line 112, in train
model.build_model()
Can you look into the issue as I am new to tensor flow,I could not point this out.

get same error. @ArjunAhuja95 , do you solve it?

line 128 and 162 in the model.py
with tf.name_scope('source_train_op')    modifies to
with tf.variable_scope('source_train_op',reuse=False):
with tf.name_scope('target_train_op')     modifes to
with tf.variable_scope('target_train_op',reuse=False):

made changes based on SrCMpink's comment, then got resource exhausted: OOM issue. Anyone know how to solve this?

@cindycfeng This code load all data into memory at once, so you need more memory to run this model

@cindycfeng Try to run it on google colabs. You will have access to more RAM and memory.


Tasks 'command' from system environment variable enclosed in single quotes

VSCode Version: 1.44.2
OS Version: Linux 5.3.11-300.fc31.x86_64 #1 SMP Tue Nov 12 19:08:07 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Steps to Reproduce:

I have a cross compile environment that sets ${CXX} to:
$ echo ${CXX}
arm-altec-linux-gnueabi-g++ -mthumb -mfpu=neon -mfloat-abi=hard -mcpu=cortex-a7 --sysroot=/home/xyz/systemroot

I am using this variable as the command in a task command in tasks.json:
"command": "${env:CXX}"

When running the task I am getting the output from the shell:

Executing task: 'arm-altec-linux-gnueabi-g++  -mthumb -mfpu=neon -mfloat-abi=hard -mcpu=cortex-a7 --sysroot=/home/xyz/systemroot' -g helloworld.cpp -o /home/xyz/vscode/helloworld/.vscode/tasks <

/bin/bash: arm-altec-linux-gnueabi-g++  -mthumb -mfpu=neon -mfloat-abi=hard -mcpu=cortex-a7 --sysroot=/home/xyz/systemroot: No such file or directory
The terminal process terminated with exit code: 127
For some reason vscode surrounds the content of the environment variable, which is the command, with single quotes before passing the entire command string to the shell. That causes the shell to look for a file that has that name when of course the first piece is the command and the rest are arguments.
Now of course I can split up the actual command and the arguments but that makes the entire thing less portable. All I want to do is source my environment and the use the environment variables in vscode.

Does this issue occur when all extensions are disabled?: Yes

Does one of the alternate quoting options solve this for you? ex:
                        "command":{
				"quoting": "escape",
				"value": "${env:CXX}"
			}


Updated Google Chrome Beta to version 103.0.5060.42.

[x] Have you signed the Contributor License Agreement?
[x] Have you checked that there aren't other open pull requests for the same manifest update/change?

[x] Does your manifest conform to the 1.0 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

@wingetbot run


Fix license compare licenses

Signed-off-by: Gary O'Neall<EMAIL_ADDRESS>

Refactors removing comment character to only be called before the license compares
Fixes a bug when a var text is followed by only white space at the end of the file
Removes beginning of line comment characters preceded by an optional tag


Not working on first time hover

Its a really good library for pop-over.
I'm using pop-over in ng-repeat (ANGULARJS). It is not working on first time hover, working fine on second time.

Sorry,I haven't tested this plugin with angularjs. Is your problem solved?

I am also facing the same issue not working for first time

you need add this code $(el).webuiPopover('show')  after  $(el).webuiPopover(options)


Update layer diamond/risk snapshot styles

Closes #82
Updated styling on mobile, centering diamond.
Updated risk snapshot hover logic, styling, and component for desktop and mobile. No longer runs off screen, opting to use shadcnui hover card component with automatic boundary styling support.

on firefox/brave on desktop the hover is cut off through the top of the page
expected behavior is for it to remain in the bounds of the visible page

passes tests on desktop and mobile letsgoooo


Registry API: TagApiSpecRevision - mixed case tags

Request:
curl --request POST 'http://x.x.x.x/v1/projects/x/apis/my-api/versions/v1/specs/proto@2d63042a:tagRevision' \
--header 'Content-Type: application/json' \
--data-raw '{
    "tag": "tagME"
}'

Response:
{
    "name": "projects/x/apis/my-api/versions/v1/specs/proto@tagME",
    "filename": "",
    "description": "",
    "revisionId": "2d63042a",
    "mimeType": "",
    "sizeBytes": 0,
    "hash": "",
    "sourceUri": "",
    "contents": "",
    "revisionTags": [],
    "labels": {},
    "annotations": {},
    "createTime": "2021-08-03T23:58:38.966919Z",
    "revisionCreateTime": "2021-08-03T23:58:38.966919Z",
    "revisionUpdateTime": "2021-08-03T23:58:38.966919Z"
}

From the response it seemed that the revision was successfully tagged but when I tried to retrieve the spec with tag, it failed with 400 Bad Request.
cur --request GET 'http://x.x.x.x/v1/projects/x/apis/my-api/versions/v1/specs/proto@tagME'

The above case works as expected if all the letters in tagme are lower cased. Seems like we don't allow mixed case for tag names. Can we reflect the same (throw an error) when calling TagApiSpecRevision?

This expression is used to validate revision tags in resource names. It seems reasonable to use it to validate tags passed to TagApiSpecRevision


How to route tab navigator from stack navigator.

I have stack and tab navigator. How to change active view on tab navigator from stack navigator?
For example:
const Root = StackNavigator({
    Home: {
        screen: ScreenOne,
    },
    List: {
        screen: ScreenTwo
    },

}, {
    });

const ListTabs = TabNavigator({
   One: {
        screen:  TabOne,
        navigationOptions: {
            title: 'Map',

        },
    },

    Two: {
        screen: TabTwo,
        navigationOptions: {
            title: 'List',
        },
    },

}, {
        swipeEnabled: false,
    });

you can do like this
const AppRouter = StackNavigator({     Root: { screen: Root},     ListTabs: { screen: ListTabs} })

@aayush-dev I tried but I have error like that. ListTabs must be a react component

How can one navigation from ScreenTwo to Tab two (List)


Connection request should allow user to specify which VLAN to be used on ingress and egress

Hi,
It is interesting that the connection request should allow the user to define which VLAN ID he/she wants to use when requesting a circuit (ingress and egress, which may be different by the way).
Choosing the user VLAN does not relate to VLAN IDs used for inter-domain connections. The latter will be automatically chosen from the SDX Controller / PCE.
In the meantime, it would be nice if the GET /connection displays which VLAN was chose for ingress and egress from the user request (again, only on user ports, not inter-OXP ports)

Hi @YufengXin can you please provide an update if the VLAN ID is already supported by SDX-Controller connection creation request?
As per the swagger API spec, it does not seems to support:
https://github.com/atlanticwave-sdx/sdx-controller/blob/main/sdx_controller/swagger/swagger.yaml#L408-L433

@italovalcy The port is an object here, which could contain the user specified VLANs.
I am sure the backend support it. However, I thought we still need to test the actual request json from you, would you please provide an example you used?
@sajith would look into it.

https://github.com/atlanticwave-sdx/pce/pull/177 ("Update getting port when breakdown") is related, right?  The PR was made when I was away, so I'm not quite up to speed on that.  I guess we should get it merged first?

Also here's #/components/schemas/port:
https://github.com/atlanticwave-sdx/sdx-controller/blob/b931068826eac56d46ec490b1826dbaecf572175/sdx_controller/swagger/swagger.yaml#L710-L743
And here's an example connection request:
{
    "id": "test-connection-request",
    "name": "Test connection request",
    "start_time": "2000-01-23T04:56:07.000Z",
    "end_time": "2000-01-23T04:56:07.000Z",
    "bandwidth_required": 10,
    "latency_required": 300,
    "egress_port": {
        "id": "urn:sdx:port:amlight.net:A1:1",
        "name": "Novi100:1",
        "node": "urn:sdx:node:amlight.net:A1",
        "status": "up"
    },
    "ingress_port": {
        "id": "urn:ogf:network:sdx:port:zaoxi:A1:2",
        "name": "Novi100:2",
        "node": "urn:ogf:network:sdx:node:zaoxi:A1",
        "status": "up"
    }
}

What would a request that carries a VLAN ID look like?  I don't believe that Topology Data Model Specification specifies a format/schema for connection requests.

@sajith I think we reused the 'label' and 'label_range' fields in the port object for the connection request, ie, 'label' for the type, vlan in this case, and 'vlan_range' for the value. See this test json in data model.
https://github.com/atlanticwave-sdx/datamodel/blob/main/src/sdx_datamodel/data/requests/test_request_p2p.json
That's why I remembered this was tested already.
@italovalcy we'll use your test request to test. But there is no specification doc for request yet. I started with the json schema like others and tried to reuse as much as possible. We can use your schema/convention.

Hi @YufengXin and @sajith I tried to use the label and vlan_range as in the data model request example above, but it didnt seems to have any effect on the sdx-controller:
curl -X POST http://<IP_ADDRESS>:8080/SDX-Controller/1.0.0/connection -H 'Content-Type: application/json' -d '{"id": "test-222-333-1", "name": "Test connection request 22", "start_time": "2000-01-23T04:56:07.000Z", "end_time": "2000-01-23T04:56:07.000Z", "bandwidth_required": 10, "latency_required": 300, "egress_port": {"id": "urn:sdx:port:tenet.ac.za:Tenet03:50", "name": "Tenet03:50", "node": "urn:sdx:port:tenet.ac.za:Tenet03", "status": "up", "label": "vlan", "label_range": "107"}, "ingress_port": {"id": "urn:sdx:port:ampath.net:Ampath3:50", "name": "Ampath3:50", "node": "urn:sdx:port:ampath.net:Ampath3", "status": "up", "label": "vlan", "label_range": "107"}}'

# Logs:

INFO:sdx_pce.topology.temanager:generate_traffic_matrix: connection_request: {'id': 'test-222-333-1', 'name': 'Test connection request 22', 'start_time': '2000-01-23T04:56:07.000Z', 'end_time': '2000-01-23T04:56:07.000Z', 'bandwidth_required': 10, 'latency_required': 300, 'egress_port': {'id': 'urn:sdx:port:tenet.ac.za:Tenet03:50', 'name': 'Tenet03:50', 'node': 'urn:sdx:port:tenet.ac.za:Tenet03', 'status': 'up', 'label': 'vlan', 'label_range': '107'}, 'ingress_port': {'id': 'urn:sdx:port:ampath.net:Ampath3:50', 'name': 'Ampath3:50', 'node': 'urn:sdx:port:ampath.net:Ampath3', 'status': 'up', 'label': 'vlan', 'label_range': '107'}}
...
INFO:sdx_pce.topology.temanager:generate_connection_breakdown(): tagged_breakdown: VlanTaggedBreakdowns(breakdowns={'urn:sdx:topology:ampath.net': VlanTaggedBreakdown(name='AMPATH_vlan_99_99', dynamic_backup_path=True, uni_a=VlanTaggedPort(tag=VlanTag(value=99, tag_type=1), port_id='urn:sdx:port:ampath.net:Ampath3:2'), uni_z=VlanTaggedPort(tag=VlanTag(value=99, tag_type=1), port_id='urn:sdx:port:ampath.net:Ampath1:40')), 'urn:sdx:topology:sax.net': VlanTaggedBreakdown(name='SAX_vlan_99_99', dynamic_backup_path=True, uni_a=VlanTaggedPort(tag=VlanTag(value=99, tag_type=1), port_id='urn:sdx:port:sax.net:Sax01:40'), uni_z=VlanTaggedPort(tag=VlanTag(value=99, tag_type=1), port_id='urn:sdx:port:sax.net:Sax01:41')), 'urn:sdx:topology:tenet.ac.za': VlanTaggedBreakdown(name='TENET_vlan_99_99', dynamic_backup_path=True, uni_a=VlanTaggedPort(tag=VlanTag(value=99, tag_type=1), port_id='urn:sdx:port:tenet.ac.za:Tenet01:41'), uni_z=VlanTaggedPort(tag=VlanTag(value=99, tag_type=1), port_id='urn:sdx:port:tenet.ac.za:Tenet03:2'))})
INFO:sdx_controller.controllers.connection_controller:place_connection result: ID: test-222-333-1 reason='Connection published', code=200

@italovalcy Thanks for testing this. could you share the OXP topology json you used? Specifically, what are the vlan_range specified in these two ports' services component?

Hi @YufengXin sure thing! Here is is:

the topology I'm using is based on the following setup (Steps to reproduce):

git clone https://github.com/atlanticwave-sdx/sdx-continuous-development
cd sdx-continuous-development
git checkout main
git pull
for repo in sdx-lc sdx-controller kytos-sdx-topology; do cd data-plane/; git clone https://github.com/atlanticwave-sdx/$repo container-$repo; cd ..; done
cd data-plane
cp template.env .env
./1_build_kytos.sh
./2_build_oxpos.sh
./3_build_local_controllers.sh
./4_build_mongo.sh
cd container-sdx-controller
docker build -t sdx-controller .
cd ..
docker compose up -d

# wait a couple of secunds to make the OXPs boot up. Usually I follow the steps below to make sure they are all UP:
EXP_SW=3; while true; do N_SW=$(curl -s http://<IP_ADDRESS>:8181/api/kytos/topology/v3/switches | jq -r '.switches[].id' | wc -l); echo "waiting switches $N_SW / $EXP_SW"; if [ $N_SW -eq $EXP_SW ]; then break; fi; sleep 1; done
EXP_SW=2; while true; do N_SW=$(curl -s http://<IP_ADDRESS>:8282/api/kytos/topology/v3/switches | jq -r '.switches[].id' | wc -l); echo "waiting switches $N_SW / $EXP_SW"; if [ $N_SW -eq $EXP_SW ]; then break; fi; sleep 1; done
EXP_SW=3; while true; do N_SW=$(curl -s http://<IP_ADDRESS>:8383/api/kytos/topology/v3/switches | jq -r '.switches[].id' | wc -l); echo "waiting switches $N_SW / $EXP_SW"; if [ $N_SW -eq $EXP_SW ]; then break; fi; sleep 1; done

# configure OXPs -- kytos requires some initial configs like approving switches and links just discovered
./container-kytos-sdx-topology/curl/2.enable_all.sh
./container-kytos-sdx-topology/curl/0a.version_control.sh
./container-kytos-sdx-topology/curl/0b.version_control.sh
./container-kytos-sdx-topology/curl/0c.version_control.sh

# check if the topology is correctly identified by SDX-Controller:
curl -s http://<IP_ADDRESS>:8080/SDX-Controller/1.0.0/topology | jq -r '.nodes[] | (.ports[] | .id)'
curl -s http://<IP_ADDRESS>:8080/SDX-Controller/1.0.0/topology | jq -r '.links[] | .id + " " + .ports[0].id + " " + .ports[1].id'

# finally I just submit a connection creation request to SDX-Controller which should span requests to all other OXPs:
$ curl -X POST http://<IP_ADDRESS>:8080/SDX-Controller/1.0.0/connection -H 'Content-Type: application/json' -d '{"id": "test-from-ampath-vlan-49-to-tenet-vlan-49", "name": "Test connection request 22", "start_time": "2000-01-23T04:56:07.000Z", "end_time": "2000-01-23T04:56:07.000Z", "bandwidth_required": 10, "latency_required": 300, "egress_port": {"id": "urn:sdx:port:tenet.ac.za:Tenet03:50", "name": "Tenet03:50", "node": "urn:sdx:port:tenet.ac.za:Tenet03", "status": "up", "label": "vlan", "label_range": "49"}, "ingress_port": {"id": "urn:sdx:port:ampath.net:Ampath3:50", "name": "Ampath3:50", "node": "urn:sdx:port:ampath.net:Ampath3", "status": "up", "label": "vlan", "label_range": "49"}}'
{
  "connection_id": "test-from-ampath-vlan-49-to-tenet-vlan-49",
  "reason": "Connection published",
  "status": "OK"
}

# get the connection recently created:
$ curl http://<IP_ADDRESS>:8080/SDX-Controller/1.0.0/connection/test-from-ampath-vlan-49-to-tenet-vlan-49
{
  "bandwidth_required": 10,
  "egress_port": {
    "id": "urn:sdx:port:tenet.ac.za:Tenet03:50",
    "label": "vlan",
    "label_range": "49",
    "name": "Tenet03:50",
    "node": "urn:sdx:port:tenet.ac.za:Tenet03",
    "status": "up"
  },
  "end_time": "2000-01-23T04:56:07.000Z",
  "id": "test-from-ampath-vlan-49-to-tenet-vlan-49",
  "ingress_port": {
    "id": "urn:sdx:port:ampath.net:Ampath3:50",
    "label": "vlan",
    "label_range": "49",
    "name": "Ampath3:50",
    "node": "urn:sdx:port:ampath.net:Ampath3",
    "status": "up"
  },
  "latency_required": 300,
  "name": "Test connection request 22",
  "start_time": "2000-01-23T04:56:07.000Z"
}

# get the circuits created on each OXP:
$ curl -s http://<IP_ADDRESS>:8181/api/kytos/mef_eline/v2/evc/ | jq -r '.[] | .id + " " + .name + " uni_a=" + (.uni_a|tostring) + " uni_z=" + (.uni_z|tostring)'
228080e6094e4b AMPATH_vlan_100_100 uni_a={"tag":{"tag_type":1,"value":100},"interface_id":"aa:00:00:00:00:00:00:03:2"} uni_z={"tag":{"tag_type":1,"value":100},"interface_id":"aa:00:00:00:00:00:00:01:40"}

$ curl -s http://<IP_ADDRESS>:8282/api/kytos/mef_eline/v2/evc/ | jq -r '.[] | .id + " " + .name + " uni_a=" + (.uni_a|tostring) + " uni_z=" + (.uni_z|tostring)'
d7ff9bbf55a94c SAX_vlan_100_100 uni_a={"tag":{"tag_type":1,"value":100},"interface_id":"dd:00:00:00:00:00:00:04:40"} uni_z={"tag":{"tag_type":1,"value":100},"interface_id":"dd:00:00:00:00:00:00:04:41"}

$ curl -s http://<IP_ADDRESS>:8383/api/kytos/mef_eline/v2/evc/ | jq -r '.[] | .id + " " + .name + " uni_a=" + (.uni_a|tostring) + " uni_z=" + (.uni_z|tostring)'
ae7b2b227c5940 TENET_vlan_100_100 uni_a={"tag":{"tag_type":1,"value":100},"interface_id":"cc:00:00:00:00:00:00:06:41"} uni_z={"tag":{"tag_type":1,"value":100},"interface_id":"cc:00:00:00:00:00:00:08:2"}

for the label range on the ports, I believe so:

$ curl -s http://<IP_ADDRESS>:8080/SDX-Controller/1.0.0/topology | jq -r '.nodes[] | (.ports[] | .id + " label_range=" + (.label_range|tostring))'
urn:sdx:port:ampath.net:Ampath3:50 label_range=["1-100"]
urn:sdx:port:ampath.net:Ampath3:2 label_range=["1-100"]
urn:sdx:port:ampath.net:Ampath3:3 label_range=["1-100"]
urn:sdx:port:ampath.net:Ampath2:1 label_range=["1-100"]
urn:sdx:port:ampath.net:Ampath2:50 label_range=["1-100"]
urn:sdx:port:ampath.net:Ampath2:40 label_range=["1-100"]
urn:sdx:port:ampath.net:Ampath2:3 label_range=["1-100"]
urn:sdx:port:ampath.net:Ampath1:1 label_range=["1-100"]
urn:sdx:port:ampath.net:Ampath1:40 label_range=["1-100"]
urn:sdx:port:ampath.net:Ampath1:2 label_range=["1-100"]
urn:sdx:port:ampath.net:Ampath1:50 label_range=["1-100"]
urn:sdx:port:sax.net:Sax01:41 label_range=["1-100"]
urn:sdx:port:sax.net:Sax01:1 label_range=["1-100"]
urn:sdx:port:sax.net:Sax01:40 label_range=["1-100"]
urn:sdx:port:sax.net:Sax01:50 label_range=["1-100"]
urn:sdx:port:sax.net:Sax02:41 label_range=["1-100"]
urn:sdx:port:sax.net:Sax02:1 label_range=["1-100"]
urn:sdx:port:sax.net:Sax02:40 label_range=["1-100"]
urn:sdx:port:sax.net:Sax02:50 label_range=["1-100"]
urn:sdx:port:tenet.ac.za:Tenet02:41 label_range=["1-100"]
urn:sdx:port:tenet.ac.za:Tenet02:1 label_range=["1-100"]
urn:sdx:port:tenet.ac.za:Tenet02:50 label_range=["1-100"]
urn:sdx:port:tenet.ac.za:Tenet03:2 label_range=["1-100"]
urn:sdx:port:tenet.ac.za:Tenet03:50 label_range=["1-100"]
urn:sdx:port:tenet.ac.za:Tenet01:41 label_range=["1-100"]
urn:sdx:port:tenet.ac.za:Tenet01:1 label_range=["1-100"]
urn:sdx:port:tenet.ac.za:Tenet01:2 label_range=["1-100"]
urn:sdx:port:tenet.ac.za:Tenet01:50 label_range=["1-100"]

@italovalcy As I can see, the vlan_range is [1-100] and you were asking vlan 107 in your request. 107 is outside of the OXP available vlan range, in which case, the current logic would just pick an available one to assign to your request.
A while ago, we agreed to retire the 'sdx-continuous-development' repo. Now you made it work, we could look at it again.
(1) could you please make thees instruction into the readme in that repo?
(2) For the timebeing, we only need to run unittest for debugging, would you be able to save these three OXP topology.json and check them in to the tests/data in 'datamodel' repo? Thanks.

@italovalcy BTW, in your testing environment, are you using the main branch for sdx-controller and sdx-lc?
If so, we'll try to reproduce your environment.  Thanks.
@mcevik0

@italovalcy As I can see, the vlan_range is [1-100] and you were asking vlan 107 in your request. 107 is outside of the OXP available vlan range, in which case, the current logic would just pick an available one to assign to your request.
A while ago, we agreed to retire the 'sdx-continuous-development' repo. Now you made it work, we could look at it again. (1) could you please make thees instruction into the readme in that repo?

Hi @YufengXin actually, if you see the requests I created two tests: 1) first using VLAN 107, which is outside of the range allowed from the OXP; 2) I created a second test using VLAN 49, inside the VLAN range and still got assigned VLAN 100....
For situation 1, the SDX-Controller has to reply saying that the requested VLAN is not allowed, instead of creating a VLAN of the choice of the SDX-COntroller.
For situation 2, the SDX-Controller just ignored the request from the user and assigned a VLAN of its choice again..

(2) For the timebeing, we only need to run unittest for debugging, would you be able to save these three OXP topology.json and check them in to the tests/data in 'datamodel' repo? Thanks.

In this case you just need to run the same steps as I did before and you should be able to get the JSONs! ;)

@italovalcy BTW, in your testing environment, are you using the main branch for sdx-controller and sdx-lc? If so, we'll try to reproduce your environment. Thanks. @mcevik0

I'm using main branch.

Thanks, @italovalcy.
I'll leave the testing environment to Mert. (I see both submodules were one month old.)
@sajith Could you please take a look at this issue? I think our unittest json tried to cover both in- and out- vlan range cases, please double check and fix. Thx.

Hi Yufeng,

I'll leave the testing environment to Mert. (I see both submodules were one month old.)

Yes, good point! Indeed, on the procedure above I just ignored the submodules and cloned directly from upstream. See the specific part here:
for repo in sdx-lc sdx-controller kytos-sdx-topology; do cd data-plane/; git clone https://github.com/atlanticwave-sdx/$repo container-$repo; cd ..; done

It would be even better if docker.io/awsdx/sdx-controller and docker.io/awsdx/sdx-lc images could be used instead of building them off submodules...
I see that using those docker images is an item in https://github.com/atlanticwave-sdx/sdx-continuous-development/issues/106.  Is the plan to resurrect that repo?  How quick/easy would it be to use it for local testing today?

Since submodules are not used anyway and the Kytos NAPP now works, let Mert and Cong try to duplicate the setup first. Then move to issue #247 on using these images.
This issue is on itself, let's get an unittest case work in duplicating Italo's test.

Hi team, any updates on this issue?

Not yet. I will get to this, but not sure when.

@italovalcy I think you relayed Jeronimo's opinion to hold on this issue, because it depends on the request specification and examples.
Anyway, I made the move, making changes  in 'datamdoel' and 'sdx-lc' first , that include cleaning up and streamline the schema and the code, after turning on the function to parse the 'vlan_range' attribute moved from the port object to the port.services.
And as I explained to you, any changes in the datamodel will trigger the chain reaction: datamodel<->sdx-lc<->pce and finally here in sdx-controller.
I have two PRs in datamodel and sdx-lc that have completed the unittest. The next step is to conduct the system tests with sdx-lc and sdx-controller before merging.


Import modules as git submodules

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@ce00efd). Click here to learn what that means.
The diff coverage is 16.66%.

@@            Coverage Diff            @@
##             master     #136   +/-   ##
=========================================
  Coverage          ?   40.08%
=========================================
  Files             ?       11
  Lines             ?      227
  Branches          ?        0
=========================================
  Hits              ?       91
  Misses            ?      136
  Partials          ?        0

Impacted Files
Coverage Δ

lib/kite/module.rb
32.14% <13.33%> (ø)

lib/kite/helpers.rb
26.66% <25%> (ø)

lib/kite/base.rb
66.66% <50%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ce00efd...f2b00c8. Read the comment docs.


Switch requested partitions from SparkContext.defaultParallelism to o…

…ne partition per 400MB.
This should work better with balanced sharding and dynamic allocation.
I believe 400MB is the correct number based on the API internals, but we could lower it to 128MB to be consistent with GCS and just let the API correct it.
I also add pretty scary log spam about the API decreasing the number of partitions, but I think this will give users clarity.
Fixes #48.

LGTM. This approach is probably better than the current approach, but it will mean that your estimation may be (significantly) inaccurate in the presence of column selection. Unfortunately, there's no good workaround for this at the moment.

@aniket486 Does this look good to you?

LGTM overall, except one thing - Let's make 400MB configurable in case it makes significantly inaccurate estimation in presence of column selection.

It already is configurable via the parallelism flag.
I can add a new parameter for the target "partitionSize" that both control the same thing, but that flag is going to be wildly off in the normal case of column or predicate filtering.


How to update black?

I just noticed my editor isn't using the latest version of black. How can I update it?

You need to modify some file codes in https://github.com/thep0y/python-black/tree/main/lib/common to update black and its dependencies.
Or you can wait for me to update it in November or December.

Consider this a feature request to allow us to update it ourselves. Maybe with an option to pin to a specific version if we want to?

I don’t have time because I’m busy at work, so I will update black when I have free time in November or December.
You can now update it yourself in the %SUBLIME_TEXT%/Packages/python-black directory.

The black package used by python-black has some modifications, which can be implemented by any user.
The black package uses the click package and some third-party dependencies. If you want to update black yourself, you need to delete the code related to the click package and update the third-party dependencies.
This is not a complicated job.
Now I have updated the black version to 21.11b1.


Is it possible to see the payload and/or the headers of requests going through the envoy?

If you are reporting any crash or any potential security issue, do not
open an issue in this repo. Please report the issue via emailing
<EMAIL_ADDRESS>where the issue will be triaged appropriately.
Title: Is it possible to see the payload and/or the headers of requests going through the envoy?
Description:

We are considering the possibility of monitoring duplicate requests between different services to identify the opportunities for caching in case some service owner does not realize it. We are thinking about grabbing the request content somewhere in our service mesh, hash it and persist it for further analysis. I am wondering whether it is possible to do so at the envoy level (We will need to access the request payload / headers to calculate the hash)? If possible, we might sample such operation at a small percentage, so hopefully, it does not impact the performance too much. Thank you!

[optional Relevant Links:]

Any extra documentation required to understand the issue.

You can configure the access log to show the header.

You can configure the access log to show the header.

But we still cannot access the request body in access log, I think? @zhangbo1882

you might also want to look at the tap filter https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/tap_filter

Thank you @phlax ! Does the tap filter output support sinking to some customized processor? We don't want to log the raw content for security and privacy reasons.

not sure i didnt use it before but my reading of the docs here is that you can either use/implement a sink or stream from an admin endpoint https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/tap/v3/common.proto#envoy-v3-api-msg-config-tap-v3-outputsink

Thank you, I will take a closer look!


Wrap poetry

So the initial idea of poe was to extend poetry to have the poetry run command.
(However poetry run does something else.)
Now this tool tries to give this missing run command, but introduces a new command instead of extending poetry.

poetry add somepackage
poe tests

What I propose is a mode (e.g. configured through the toml) which wraps poetry to have it a single executable to remember.

poe add somepackage
poe run tests

Hi @luckydonald, thanks for the idea. I guess what you mean is that the poe CLI would by default behave like an alias for the poetry CLI, but if you define poe tasks then they're added to the tasks available in poetry?
This sounds almost like what you get if you install the poethepoet poetry plugin and configure tool.poe.poetry_command = "". In this configuration poe tasks are added directly to the poetry CLI. Have you tried that?

Ah, I tried installing the poetry as plugin and that didn't quite work, I skipped that part because it was so confusing, and seemed like an extra, and I didn't understand how it would work like you just explained. Though I was searching for that.
Especially as I came from a stack overflow page where you advertised your plugin as a solution.

Whatever tool.poe.poetry_command is set to must not already exist as a poetry command!
What would happen then?


chore: fixing issue with biztalk360

Summary
SImple fix for biztalk ID

I think there is a problem with the check here. It's saying that the display name is already taken, which not really true. I'm just trying to change the id because I have to reference it in the marketplace.
ERROR: The following errors occurred while validating: biztalks360

dataSourceMetadata/displayName: displayName has already been taken

@rahulbasu-nr , I'm not sure if you know the answer to this one. Someone put the id for the datasource here as third-party-biztalk360, and I'm trying to fix it into something more usable for the marketplace. Is it possible to change the ID of a data-source once it's been set? And if so, how can I resolve the schema compliance error?


[SPARK-45845][SS][UI] Add number of evicted state rows to streaming UI

What changes were proposed in this pull request?

In a stateful query, a watermark has two responsibilities:

drop late rows
Evict state rows from state store

Before we only log "aggregated number of rows dropped by watermark". This is case 1. But people would confuse this with case 2. This PR purpose we also add a chart for case 2. Also made the explanation for case 1 more verbose.
Now:

Why are the changes needed?

UI improvement
Does this PR introduce any user-facing change?

Yes, users will be able to view the new UI
How was this patch tested?

Manually tested
Was this patch authored or co-authored using generative AI tooling?

No

cc @HeartSaVioR  can you take a look please : )

https://github.com/WweiL/oss-spark/actions/runs/6829179497/job/18579693008
Could you please check with UISeleniumSuite and confirm that it's not related to this change?

@HeartSaVioR Thanks for pointing it out! Yes the two UISeleniumSuite related ones are indeed caused by this PR. I verified locally that they don't fail after my new commit

Thanks! Merging to master.


ISSUE-169: GOENV_GOPATH_PREFIX does not work as expected

When rehash is executed and GOENV_DISABLE_GOPATH is 0 (default)
the code should use currentVersionName instead becasue GOENV_VERSION is
not set.

Ok, the latest commit eventually fixed the OSX build too.


Add many details and DEVICE_ID peripheral to STM32F429

Huge parts are not by me. Copying authorized in:
https://github.com/jmesmon/stm32f429/pull/3#issuecomment-414151828

Sorry, quite a lot of time has passed on this @astro.  The ST license does not appear to prohibit making changes.  To keep the vendor provided ST clear from the patched can you submit this as a separate file named {name}.patched.svd?  Thanks!


Change how the uploaders throw errors

See comment. The validation steps in the uploaders should return the list of errors and then the upload() method should check the result of validation and throw the error.

I think this is done.


How is NotUntil<P> supposed to be used?

I was trying to use this crate to implement rate limiting in a synchronous context, and using the default direct rate limiter, where a negative decision hands back NotUntil<P> so the caller can interrogate when a conforming decision is likely to be able to be made.
OK, cool... and there's NotUntil::earliest_possible and NotUntil::wait_time_from, but as it turns out, they're useless from outside of RateLimiter itself unless you also manage a clock yourself. NotUntil::earliest_possible gives you a quasi-Instant which needs another quasi-Instant from which to derive a Duration of how long to wait, and likewise, NotUntil::wait_time_from needs the same thing. RateLimiter does this by just calling self.clock.now() to get that from parameter required for NotUntil::wait_time_from... but there's no way to actually access the clock being used by RateLimiter.
Thus, in the end, you have to manage your own clock and keep it side-by-side with RateLimiter just to do anything in a synchronous context, which seems like a big ergonomics pitfall.

These are great questions - that this is unclear at all is a docs bug, at least.
The main method that code is supposed to use is earliest_possible - the method is meant to return the timestamp you need, and that timestamp should be the type that corresponds to the Clock in use. From what you're writing, it sounds like that is not the case for you?
E.g., in this code:
#[test]
fn default_direct() {
    let clock = governor::clock::DefaultClock::default();
    let limiter: DefaultDirectRateLimiter =
        RateLimiter::direct_with_clock(Quota::per_second(nonzero!(20u32)), clock);
    match limiter.check() {
        Ok(_) => (),
        Err(nu) => { // here, `nu.earliest_possible()` returns a `Quanta::Instant`.
        }
    }
}

You're right that you need a quanta clock structure with the default clock. The secret that the docs don't tell you (but should!) is that governor only tells a difference in concrete instances any Clock in the FakeRelativeClock structure (for testing) and in QuantaUpkeepClock (for more coarse-grained timings where speed really really matters). Approximately all the other clocks accessible by default implement Default::default so you can get a clock instance if you should need it.
If all that is too annoying to manage and you don't need the microseconds of latency per measurement, you can use a MonotonicClock rate limiter, which uses std::time::Instant and not worry about managing the clock at all... if you disable the quanta feature in the crate, that should be the default clock, also.
Please let me know if that helps!

Yeah, this is what I ended up doing for now... but my complaint is really more about the ergonomics of it all.
It seems like RateLimiter ought to at least expose a reference to the clock it's using, or alternatively, change NotUntil::wait_time_from to be like.. NotUntil::earliest_possible_duration? which takes no arguments and just requires that the middleware creating it passes in the current time so it can do the math for you without having to manually deal with the clock at all.
Essentially, I expect NotUntil to be able to just give me a Duration that I should wait before trying again, without having to think about the clock at all.

Yep, that's fully valid. I just realized that back in April I merged #232, which adds a RateLimiter::clock method but didn't cut a release! Massive oops on my part.
I'll get the release cut now, hope that helps with your immediate issue at least.

OK, 0.6.4 is out now with a clock method on RateLimiter (and now I remember what kept up that release, the gh action to do so had a bug).
I'll leave this issue open - feel free to post the code you end up with, I'd love to integrate that in an example.

👍🏻
I ended up removing the rate limiting code to simply some things during development, but I'll definitely swing back around and try things out with 0.6.4. I had also encountered some issues with limiting behavior at higher throughput rates, so I might have another issue to open. 😅

Note that #245 exposed a semver incompatibility: I had to yank 0.6.4 and re-release the change as 0.7.0. No semantic difference except in version number (:


Kubernetes discovery support

Given the container/cloud-oriented nature of Crate and Kubernetes, it would be great to support Crate discovery on Kubernetes clusters.
Elasticsearch already supports it, but it seems #995 is a blocker.
According to your people, there is a solution planned somewhere in your backlog, so I'm creating this issue to keep up-to-date on that subject.

hi @pires
service discovery via DNS is now implemented, currently on master.
here's the corresponding PR https://github.com/crate/crate/pull/2067
best,
christian

DNS is not Kubernetes but OK.
On May 27, 2015 07:55, "Christian Haudum"<EMAIL_ADDRESS>wrote:

Closed #1728 https://github.com/crate/crate/issues/1728.
—
Reply to this email directly or view it on GitHub
https://github.com/crate/crate/issues/1728#event-314543219.

yes, you're right, this is not kubernetes discovery (via API)
however, DNS is one possible way of node discovery on kubernetes clusters.

DNS is still an add-on and not a first-class citizen of Kubernetes. However, if you rely on Elasticsearch discovery it's easy to implement. See https://github.com/fabric8io/elasticsearch-cloud-kubernetes, it supports DNS as well.

By the way, kubernetes has an example with elasticsearch_discovery.go plugin that dynamically generates config for elasticsearch and adds discovered hosts https://github.com/kubernetes/kubernetes/tree/master/examples/elasticsearch

That is being deprecated.
On Aug 25, 2015 00:25, "Jaka Hudoklin"<EMAIL_ADDRESS>wrote:

By the way, kubernetes has an example with elasticsearch_discovery.go
plugin that dynamically generates config for elasticsearch and adds
discovered hosts
https://github.com/kubernetes/kubernetes/tree/master/examples/elasticsearch
—
Reply to this email directly or view it on GitHub
https://github.com/crate/crate/issues/1728#issuecomment-134411740.

@pires

As of Kubernetes 1.3, DNS is a built-in service launched automatically using the addon manager cluster add-on. A DNS Pod and Service will be scheduled on the cluster, and the kubelets will be configured to tell individual containers to use the DNS Service’s IP to resolve DNS names.
(source: http://kubernetes.io/docs/admin/dns/)

You can find a template utilizing DNS discovery for a Crate cluster here: https://crate.io/a/kubernetes-and-crate/

Thanks. I'm closing this because I moved forward.


"Find" locates 4 matches, but is unable to scroll to them.

My issue is with the find function on the mac. Before installing the theme I could find al the matches of a string and flip through the document. Now when I search a document which for example has 4 matches it displays "4 matches, unable to find matches" in the left-hand bottom corner. It encircles the matches as I scroll through the document, but I used to be able to just hit Enter multiple times to flip through the matches and sublime would scroll through the document to display. Besides that, I think you did an awesome job on the UI!

Can't reproduce. A theme affecting how search functionality works seems rather unlikely. Maybe it was a ST bug, which is now fixed.


BasemapGallery tooltips: enable links and adjust timing

Html links included in tooltip's in the Basemap Gallery cannot be clicked. This PR adds a parser to remove any html tags [e.g. <html> and </html>] from tooltip text. This means the text appears as plain text and does not appear as a link.

Is it possible to use something like onLinkActivated: Qt.openUrlExternally(link)
https://stackoverflow.com/questions/12536416/qml-text-element-hyperlink

Thanks for your comments @anmacdonald and @ldanzinger.
Using a Connection we have managed to use the Qt.openUrlExternally(link) method to enable the link in the tooltip to be opened.
With this change, the timing of the tooltip needed to be modified.  Currently, if the mouse leaves the thumbnail MouseArea, the tooltip will disappear. This means that a user cannot click the link in the tooltip because the tooltip disappears. Two properties are available with attached tooltips: delay and timeout, however, the timeout duration appears to be from the time the tooltip appears rather than the time the mouse leaves the MouseArea and is not suitable for this. As such, delays to the appearance and disappearance of the tooltips were manually created using two Timer objects and the onEntered and onExited slots of the MouseArea object.
Please could you review these updated changes?
NOTE: I have modified the title of the PR from Remove html links from BasemapGallery tooltips to BasemapGallery tooltips: enable links and adjust timing. This is to reflect the changes to the purpose of the work.

Thanks for your feedback @anmacdonald and for your assistance making the changes. The changes are ready for re-review.


CacheManager.Exists is flaky

Hi, I am using the following configuration to build the cache manager.
        var cache = CacheFactory.Build("CacheManager", settings =>
        {
                var serializerSettings = new JsonSerializerSettings { ObjectCreationHandling = ObjectCreationHandling.Replace, TypeNameHandling = TypeNameHandling.Auto };
                settings = settings.WithUpdateMode(CacheUpdateMode.Up)
                                   .WithGzJsonSerializer(serializerSettings, serializerSettings);

            var part = settings.WithMicrosoftMemoryCacheHandle()
                               .WithExpiration(MapInternalExpirationMode(ExpirationMode), TimeSpan.FromMinutes(ItemTimeout));

            part.And
            .WithRedisConfiguration("Redis", config => config.WithAllowAdmin().WithDatabase(0).WithEndpoint("localhost" , 6379))
            .WithMaxRetries(50)
            .WithRetryTimeout(100)
            .WithRedisBackplane("Redis")
            .WithRedisCacheHandle("Redis")
            .WithExpiration(MapInternalExpirationMode(CacheItemExpirationMode.Sliding), TimeSpan.FromMinutes(30));
        });

Then I use cache.Put(key, value); to set a value in the cache.
Then I retrieve a memory cache handle and a redis cache handle using the following code.
var localHandle = (MemoryCacheHandle)_cache.CacheHandles.First(h => h is MemoryCacheHandle);
var redisHandle = (RedisCacheHandle)_cache.CacheHandles.First(h => h is RedisCacheHandle);
The issue is when I am calling the redisCache.Exists(key). It returns true sometimes and false intermittently  which is puzzling since the memory cache handle is consistent all the time. Any pointers as to what I might be doing wrong?

Redis might have removed they key. That can be due to your expiration settings or memory pressure?
Exists also doesn't "touch" the key, which means, if you don't do a GET, the sliding expiration window will not get extended and the key expires...
I'm pretty sure the exists method works fine. If you still think there is a bug or an issue, please try to give me an example to easily reproduce the behavior ;)


Next Steps

Let's try to write out all the next steps to get the website up and running and assign them to people so that we can avoid overlap.
Content
For more details please refer to this earlier thread about the taxonomy. https://github.com/nodejs/new.nodejs.org/issues/19#issuecomment-125007795

[ ] Home
[x] About.md
[x] Security.md
[ ] Downloads.md
[x] Foundation.md

[x] members.md
[x] TSC.md
[x] TSC meeting minutes

[ ] Community.md
[ ] Resources.md

[ ] FAQ.md
[ ] ES6.md

[x] API (link to https://iojs.org/api/ for now)
[ ] Blog.md

Templates

[x] About.hbs
[x] Security.hbs
[ ] Downloads.hbs
[x] Foundation.hbs
[ ] Community.hbs
[ ] Resources.hbs
[ ] Blog.hbs

Style Fixes

[ ] Home
[ ] Blog
[ ] General style for all the basic markdown pages like About and Foundation.

Blog

[x] Merge in old io.js posts from the evangelism repo. (Mikeal is working on this)

RSS feeds

[x] Blog posts
[x] TSC meeting minutes

@fhemberger can you edit the checklist add any additional work.

By the additional .md files, are they content writing? I'm able to help in any way I can if the quality of the weekly updates I churned out are satisfactory.

@diagramatics Uhm, at least I used the checkboxes for content migration from nodejs.org/iojs.org, not necessarily for content (re)write or editing, I think this will be the next step.

@diagramatics @fhemberger Corrent, with the exception of the Resources page which I think should be something new and Docs WG should decide/write that.

So this whole things isn't about making a real new site (including a different design), it's just about merging the contents of the io- and node-site, right?
Does this mean that a re-design would be completely inappropriate? Or at least suggestions for a new layout? Because if I take a look at the current state, nothing has changed regarding styles.

@leo A little bit of everything. ;) First of all, with the launch of the converged node/iojs v4 being scheduled for October, we want a unified website in place with well structured and up-to-date content. So this is what we're focussing on at the moment. Getting everything set-up (also for the localization teams to start), optimize the site's structure and revise the content.
During the process, we're also cleaning up the HTML and CSS. Design improvements are of cause welcome, but I guess for the moment we won't pursue a radical redesign.

Ahh, okay!
Then I'll have a look into what needs to be done there and check back again when the re-design happens (since I'd like to help there in terms of development or appearance).

Added Governance, Working Groups, FAQ, ES6 and a 404 page.

I added a rough start to the downloads page (iojs skewed right now) as well as starting to port over the i18n and linking handlebar helpers which were helpful in earlier i18n work on iojs.org. There's also some proof of concept work around a "download .md from external repo folders" hiding in the project now.

Hey @rvagg and @nodejs/build are we going to mess up all the current download links in the migration to the new site if we don't have a new download service solution worked out?

I just pushed a few big changes to the nav and footer.

Moved "Resources" to "Docs" and put "API Docs" under "Docs"
We have way too much stuff in the top nav.
Having one area with docs called "Resources" next to another called "API Docs" was terribly confusing.
We can debate names all day but people looking for docs tend to look for the word "Docs" or "Documentation"
Moved "/community" to "/contribute"
The entire project is built by the "community." Each section should have information about how the community maintains those resources.
The text I used for the top nav is "Get Involved" which is nice and action oriented.
Removed bottom nav.
This nav was largely redundant and hardly anyone gets to the bottom anyway.
We'll probably need this space for the localizations at some point.

Should we have a top level section called "Help"?
I'm not getting notifications for the all the form submissions for joining the foundation and there is a surprising number of support requests that come through simply because it's the only resources on the site that is a form.

Not sure exactly what the question here is re downloads but here's my current thoughts on the matter:

Move to download directory structure in use on iojs.org but keep it backward compatible like we do there, which means a /download/ directory to contain the different download types with /dist/ being a symlink to /download/release/ so it's basically like nodejs.org/dist
Make io.js releases available in /download/io.js/ where that is what we have now in iojs.org/download, also ensure that iojs.org/download/ and iojs.org/dist continue to work as they do now with no interruption.
Nothing breaks for existing releases, future releases follow roughly the same pattern in nodejs.org/dist so most existing expectations and assumptions still hold.
When we move to a CDN for distributing, we'll just have to adapt over the top of this but we should prioritise backward-compatibility and introducing a CDN into the mix right now is just going to slow us down so we should push that out further.

Clear as mud?

@rvagg our infra has the io.js releases, obviously, but do we have all the old node.js releases as well so that we don't break all the links to nodejs.org releases the moment we go live with the new site?

@mikeal working on new infra atm, assume this is going to go live on different infra than we are using for iojs.org right now. Will make sure that all binaries are available so that nodejs.org feels the same to users.
One side item—can we have a catalogue of links from nodejs.org that are going to break as a result of this upgrade (if any) so we can add in redirects to the server config to avoid 404s please.

One side item—can we have a catalogue of links from nodejs.org that are going to break as a result of this upgrade (if any) so we can add in redirects to the server config to avoid 404s please.

Very good point, we should put that on the checklist.

Speaking of downloads. In trying to build the download/release pages (as in user friendly ones) we'll need a version of https://iojs.org/dist/index.json which works for all Node versions, past-and-future. I'm okay with it being two files -- one for the limited io.js line and one for Node -- as long as we can download them as part of the new.nodejs.org build process.

@snostorm I can make that happen when we get new infra for this. Not going to touch the old infra though.

@rvagg works for me :) Just wanted to mention the gap

One side item—can we have a catalogue of links from nodejs.org that are going to break as a result of this upgrade (if any) so we can add in redirects to the server config to avoid 404s please.

@rvagg Not complete yet, but should catch most of the cases:
rewrite ^/about/advisory-board/                                                     https://$server_name/about/organization/ permanent;
rewrite ^/about/advisory-board/members/                                             https://$server_name/about/organization/ permanent;
rewrite ^/about/organization/tsc-meetings/                                          https://$server_name/foundation/tsc/minutes/ permanent;
rewrite ^/about/organization/tsc-meetings/(.*?)/minutes.html                        https://$server_name/foundation/tsc/minutes/$1/ permanent;
rewrite ^/about/security/                                                           https://$server_name/security/ permanent;
rewrite ^/advisory-board/(.*)                                                       https://$server_name/about/organization/ permanent;
rewrite ^/contribute/                                                               ???? permanent;
rewrite ^/contribute/accepting_contributions.html                                   ???? permanent;
rewrite ^/contribute/becoming_collaborator.html                                     ???? permanent;
rewrite ^/contribute/code_contributions/                                            ???? permanent;
rewrite ^/contribute/code_contributions/workflow.html                               ???? permanent;
rewrite ^/documentation/                                                            ???? permanent;
rewrite ^/documentation/localization/                                               ???? permanent;
rewrite ^/documentation/tutorials/                                                  ???? permanent;
rewrite ^/foundation/blog.html                                                      https://$server_name/blog/ permanent;
rewrite ^/foundation/members.html                                                   https://$server_name/foundation/members/ permanent;
rewrite ^/images/foundation-visual-guidelines.pdf                                   https://$server_name/static/documents/foundation-visual-guidelines.pdf permanent;
rewrite ^/images/logos/js-black(.*)                                                 https://$server_name/static/images/logos/js-black$1 permanent;
rewrite ^/images/logos/nodejs-(.*)                                                  https://$server_name/static/images/logos/nodejs-$1 permanent;
rewrite ^/images/node-foundation-by-laws.pdf                                        https://$server_name/static/documents/node-foundation-by-laws.pdf permanent;
rewrite ^/images/.*trademark-policy.pdf                                             https://$server_name/static/documents/trademark-policy.pdf permanent;
rewrite ^/video/(.*)                                                                https://$server_name/static/video/$1 permanent;
http://blog.nodejs.org/\d{4}/\d{2}/\d{4}/node-(v\d{1,2}\.\d{1,2}\.\d{1,2})(.*)/$    https://$server_name/blog/release/$1/ permanent;
http://blog.nodejs.org/(.*)                                                         https://$server_name/blog/$1/ permanent;

Weekly news link is now automatically generated from blog/weekly-updates/*.md, please make sure to always add the variable date: YYYY-MM-DD to the YAML frontmatter for this to work.

By the way: I have access to nodejs.org, so if we need a preview site, maybe I can set something up (preferably with a GitHub hook).

http://new-nodejs.iojs.org/ is "working" now, I just need to make some small tweaks to get the build steps to actually publish what's in ./dist whenever we push (right now it runs a placeholder script to match io.js)
Might not be worth hacking in to to the existing nodejs.org server as I believe @nodejs/build is winding up new servers to tie in to https://nodejs.org/dist/ for the 4.0 release.

@snostorm Ah, okay.

Are the initial tasks up to date?
According to previous comments by @mikeal it looks like a couple of sections has been renamed

Community -> Contribute
Resources -> Docs

And contribute and downloads was closed with #37 and #35?

@phillipj renamed in the checklist :)

I still would like to have the download-matrix from nodejs.org for the new site as well:

Also, what about additional platform builds? @mikeal @rvagg Do you know more about it?

No, I took the basic visual design from nodejs.org, but started with fresh html/css. Assets have moved from /images to /static. If you need something from the old site, just copy it from nodejs.org for now.

status update on the @nodejs/build end:

I've provisioned a new.nodejs.org and DNS is pointing to it, I've begun writing an ansible script to reconcile the setup from old & new and there some issues that I'll file separately for discussion, I'll also integrate @fhemberger's redirects there and it can all be edited via github by both website and build teams.
We're working with CloudFlare to experiment with having them front the web server and handle the bulk of the load
I just (finally!) got new wildcard SSL certs for *.iojs.org and *.nodejs.org owned by the Foundation, they are in my inbox but I haven't looked yet.

Speaking of downloads and versions.. Should the site contain both io.js and node.js or just (the converged) node.js?

Both. There still needs to be some work done to integrate the builds automatically into the site.

So, folks, you only have a temporary file in ./public/ which is what's copied into the live directory at build time, everything else is living in the root, can we come to an agreement about what actually gets published to the site and what directories to copy and what to not copy? It's doing a node_modules/.bin/gulp build at the moment, is this the wrong thing to be doing?

AFAIK @snostorm copied those parts from the old website. The site is now using npm run build (which is running the build.js command in the root folder) to generate all files into build.

ya, everything is in ./build, even static files get copied in there. the test server just serves that up (although we'll need to add a few routing rules soon).

we have had some good progress today, I'm considering closing this Issue and filing a new one with a much more specific TODO list that we need to crank out in order to be live on time.

OK, so, you want to npm run build and then rsync the contents of ./build into public?

Site is live now @ https://new.nodejs.org/en/index.html
You have to use https for this or you'll be redirected to nodejs.org
There is no index.html in the root so you'll get a Forbidden from that, probably something you need to fix?
Config is evolving @ https://github.com/nodejs/build/pull/162/files (setup/www/resources/nodejs.org)
I've set up the github-webhook for this on that server so pushes to master should result in a build in the same way as on iojs.org although it does npm run build and copies the contents ./build/ to the public directory.

Sorry, I should have waited, I'm going to wipe the server and run my script again to make sure it's complete so far.
Once we have confidence in the basics of the Ansible config then I might do an rsync of nodejs.org/dist/ and make it available to you there so we can move closer to live state. I'm also going to turn on CloudFlare's CDN for new.nodejs.org as soon as I'm able (hopefully before the weekend).

There is no index.html in the root so you'll get a Forbidden from that, probably something you need to fix?
Can you please add a hard redirect in nginx.conf from / and index.html to /en/? Its way better than adding empty index.html files with meta redirects.

ah, right, will do, I thought index.html was doing something fancy with language selection

No, but there's a separate nginx module allowing redirects based on the user's Accept-Language header. Maybe we can use this in the future.

Closing this in favor of https://github.com/nodejs/new.nodejs.org/issues/58


Waitbeforeclick not waiting

browser.ClickButton("Search", new Options { WaitBeforeClick = TimeSpan.FromMilliseconds(0.2) } )
This doesn't seem to work for me - I've increased it to 20 seconds but it is definitely not pausing before clicking, I've looked in the source code and I can't see anything crazy in there that means it shouldn't wait?

fixed in https://github.com/featurist/coypu/pull/191

Coypu 3.0.0-rc has been released https://github.com/featurist/coypu/releases/tag/3.0.0-rc


Add specific use cases to the readme

#483  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @thisisgoldman and the rest of your teammates on  Graphite


Add show_only_folders property

Hi @crahan
I've added show_only_folders property to FileChooser class.
It may be useful in cases when the user is asked to select a folder (ie output folder for saving multiple generated files).
Please let me know if you have any comments.
Thanks.
Andrii

Thank you for your contribution!
The PR looks good. I'm wondering about the risk of overwriting files that already exist but which are hidden from sight when using this option though. Currently the widget allows for selecting a folder by navigating into it and leaving the filename blank. Is that option too cluttered? Do you consider not showing the files that might be overwritten as an issue (not giving the user a visual cue that a folder they navigated into does have files inside it)?
There's also the risk of users still entering a filename which might make things confusing. The easiest solution would be to disable the filename input field when the file dialog is created or disable and clear the field when the flag is set on an existing file dialog. I would need to think about that option a little more though.

Thanks for a quick response.
On the project I'm working on overwriting existing files is the intended behavior. For other use cases we can highlight selected folder with a different color (as it's currently done for files). Let me know if you like this idea.
I think leaving filename blank is ok when the user just wants to select existing folder.
When they do type something in that field I would interpret that as a need to create a new folder.

Hi @crahan
I went ahead and added highlighting existing folder selection.
Let me know if you think other changes are needed.

The option to enter a folder name and have it highlighted (to indicate a folder already exists) would cause confusion if there's already a file with the same name (which would not be visible if only folders are being shown). I'll need some time to review, but at this point I'm leaning more towards implementing the folder-only option in such a way that the text input field for entering a file name is disabled or removed completely when only folders need to be shown.

I still think that having that field enabled may be useful in some cases (ie giving the user option to select a name for a new folder). But I agree that the case you described (user entered folder name = existing file name) can cause confusion.
One possible solution would be to highlight such case (user entered folder name that matches existing file name) with a different color (ie 'red').
Another solution - do not allow to close the dialog until a 'proper' folder is selected (the one with the name that doesn't match any existing files).
But if you insist on having that field disabled completely - that's ok. Let me know - I can implement either option. Thanks.

To not complicate things unnecessarily I'd prefer the option to disable the input field when folder-only mode is selected. In that case the dialog becomes a folder chooser.

Sounds good, thanks.
I just disabled this field for show_only_folders mode. Let me know if you have any other feedback.

Thanks for the updates. I'll review and try and get this merged asap.

I just wanted it to be more consistent with file selection version. I can remove it if you think it's cleaner that way.

Ok let's leave it in for now and I'll try and review asap. Thanks for the help!

I've merged this into a new dev branch to track improvements before the next release. Thanks for the contribution!

@andriykorchak would you mind testing the current version in the dev branch? I've added some functionality in addition to your changes for the folder-only mode:

moving between folder-only and standard mode now hides/shows the filename field (and expands the path selector accordingly)
going to folder-only mode resets the dialog (i.e. unsets any selected values) and zeros out the filename value
going to standard mode resets the dialog (i.e. unsets any selected values) and sets the filename to the default value

I've run some initial tests and everything appears to work as expected.

Sure, will do.

@crahan I've tested dev branch and everything works for me.
Unrelated question: I was planning to implement a suggestion from #12 (filter file names). Should I create a pull request to dev branch or it's better to still use master?
Thanks.

Dev is probably best, thanks!


Add dependabot.yml for Dockerfile versions

Configures GitHub to find out of date image versions in Dockerfiles.
(Originally had the Dockerfile changes, too.)

Dependabot does not do the full version update that we need -- if you update the base OS image, you also need to update the package versions that are hardwired. But several Eve packages do not hardwire the package versions, so we might as well take advantage of Dependabot, since it is built into GitHub.

Dependabot does not do the full version update that we need -- if you update the base OS image, you also need to update the package versions that are hardwired. But several Eve packages do not hardwire the package versions, so we might as well take advantage of Dependabot, since it is built into GitHub.

I recommend having this .github/dependabot.yml file checked in, but not enabling Dependabot in the reference lf-edge tree. It can then be used in any fork where it can be reviewed and tweaked before the user submits the PR as I am doing here.
I enabled Dependabot in my fork by enabling it in Settings → Security & analysis:
https://github.com/dkionka/eve/settings/security_analysis
Then I open: Insights → Dependency graph → Dependabot
https://github.com/dkionka/eve/network/updates
Every time I update the .yml it re-runs the analysis. If a version needs to be updated it generates a PR:
https://github.com/dkionka/eve/pulls?q=
Those PRs are on branches that you can find with:

$ git branch -a | grep dependabot/
remotes/origin/dependabot/docker/pkg/alpine/lfedge/eve-alpine-5.19.0
remotes/origin/dependabot/docker/pkg/guacd/alpine-3.13.0
remotes/origin/dependabot/docker/pkg/qrexec-dom0/alpine-3.13.0
remotes/origin/dependabot/docker/pkg/qrexec-lib/alpine-3.13.0
remotes/origin/dependabot/docker/pkg/watchdog/alpine-3.13.0
remotes/origin/dependabot/docker/pkg/wlan/alpine-3.13.0
remotes/origin/dependabot/docker/pkg/wwan/alpine-3.13.0
remotes/origin/dependabot/docker/pkg/xen/alpine-3.13.0

I then cherry-pick from all the branches and run a local test build. Finally I squash all of the cherry-picks and create my own PR for lf-edge.

I recommend having this .github/dependabot.yml file checked in, but not enabling Dependabot in the reference lf-edge tree. It can then be used in any fork where it can be reviewed and tweaked before the user submits the PR as I am doing here.
I enabled Dependabot in my fork by enabling it in Settings → Security & analysis:
https://github.com/dkionka/eve/settings/security_analysis
Then I open: Insights → Dependency graph → Dependabot
https://github.com/dkionka/eve/network/updates
Every time I update the .yml it re-runs the analysis. If a version needs to be updated it generates a PR:
https://github.com/dkionka/eve/pulls?q=
Those PRs are on branches that you can find with:

$ git branch -a | grep dependabot/
remotes/origin/dependabot/docker/pkg/alpine/lfedge/eve-alpine-5.19.0
remotes/origin/dependabot/docker/pkg/guacd/alpine-3.13.0
remotes/origin/dependabot/docker/pkg/qrexec-dom0/alpine-3.13.0
remotes/origin/dependabot/docker/pkg/qrexec-lib/alpine-3.13.0
remotes/origin/dependabot/docker/pkg/watchdog/alpine-3.13.0
remotes/origin/dependabot/docker/pkg/wlan/alpine-3.13.0
remotes/origin/dependabot/docker/pkg/wwan/alpine-3.13.0
remotes/origin/dependabot/docker/pkg/xen/alpine-3.13.0

I then cherry-pick from all the branches and run a local test build. Finally I squash all of the cherry-picks and create my own PR for lf-edge.

I made this the .yml only change, since I explain how to use it above, and I moved the Dockerfile changes to: https://github.com/lf-edge/eve/pull/1817

I made this the .yml only change, since I explain how to use it above, and I moved the Dockerfile changes to: https://github.com/lf-edge/eve/pull/1817

Awesome @dkionka -- one last thing -- can you please move the explanation you provided in a PR to be a section in this doc https://github.com/lf-edge/eve/blob/master/docs/CI-CD.md -- just add it at the very bottom, ok?

Awesome @dkionka -- one last thing -- can you please move the explanation you provided in a PR to be a section in this doc https://github.com/lf-edge/eve/blob/master/docs/CI-CD.md -- just add it at the very bottom, ok?

Awesome @dkionka -- one last thing -- can you please move the explanation you provided in a PR to be a section in this doc https://github.com/lf-edge/eve/blob/master/docs/CI-CD.md -- just add it at the very bottom, ok?

I moved it to to the doc.
I made it more generic because I am also working on scripts to list package versions to change, and that will be a 2nd section under maintaining Dockerfiles.

Awesome @dkionka -- one last thing -- can you please move the explanation you provided in a PR to be a section in this doc https://github.com/lf-edge/eve/blob/master/docs/CI-CD.md -- just add it at the very bottom, ok?

I moved it to to the doc.
I made it more generic because I am also working on scripts to list package versions to change, and that will be a 2nd section under maintaining Dockerfiles.

This is the followup change to add version check scripts: https://github.com/lf-edge/eve/pull/1825

This is the followup change to add version check scripts: https://github.com/lf-edge/eve/pull/1825

I fixed the yetus errors, so this should be ready to be merged.

I fixed the yetus errors, so this should be ready to be merged.

I thought that this would stay unused unless you enabled dependabot, but it is creating branches and generating PRs like https://github.com/lf-edge/eve/pull/1847 .

That's fine -- we may as well turn it on -- the bigger question is coordinating flag days for upgrades @dkionka


Chromium dev and beta might be vulnerable to CVE-2019-5786

Issue description
I spent some time trying to figure out whether dev and beta are vulnerable. It seems like the commit which fixes the vulnerability is this one:
https://github.com/chromium/chromium/commit/ba9748e78ec7e9c0d594e7edf7b2c07ea2a90449 (for dev)
https://github.com/chromium/chromium/commit/0b8ac062693ce67019dfef28f76e0c79db8fa0a3 (for beta)
The first dev tag is 74.0.3721.0, the version I see in nixpkgs is 74.0.3717.0
The first beta tag is 73.0.3683.60, the version I see in nixpkgs is 73.0.3683.56
@bendlas @Infinisil @ivan
Steps to reproduce
Technical details
Please run nix-shell -p nix-info --run "nix-info -m" and paste the
results.

has this been resolved with https://github.com/NixOS/nixpkgs/issues/56998 et al?


Error

Hi I always got this error when try to enable flavors:
awk: unknown option -i ignored

awk: can't open file /.*\/\/\/ <flavor-products>/ {
            while (getline line < "/tmp/flutter-flavor-android.part") {
                print line
            }
        }
        //
 source line number 1

I guess that you are using Linux.
See limitations on README.md, sed and ask commands don’t work the same way on Linux and macOS.
Only macOS has been tested yet.

Absolutelly not :)
I use MacOS

@sestegra please help me, I've a big problem with flavors, I try to handle all manually but with many problems when I try to debug a flavor, please if you can check you git  I'm very happy to help you to implement some features. Regards

I found the issue. On my computer, I'm using GNU awk version.
To fix your issue,

Install gawk brew install gawk
Replace awk by gawk in flutter-flavor script

Thanks now it's work!!


Closing this in favor of opening a new one.


WebM videos don't play in Safari browser (use ogv.js?)

Ah, the fun fun world of video formats. :)
Uploaded WebM or Ogg videos such as on https://lib.reviews/team/developers/post/b0873e50-4830-4603-a765-2f1c16ac3029 don't play as-is on Safari (or on current versions of Edge, though Edge is adding support, currently installable via plugin).

There are basically two ways to handle this: transcode to MP4 on the backend (may expose larger licensing issues), or to play them in non-supporting browsers via ogv.js by dynamically swapping out the video or audio element for an OGVPlayer instance.
One potential issue with ogv.js -- currently it looks like you're using the default browser controls on the playback element, and ogv.js doesn't by default ship with control support. I can fix that though. Alternate: use something like video.js as an intermediary.

Yeah, I figured it wouldn't work in those browsers yet - thanks for taking the time to write up the issues! My thinking right now is to do server-side transcodes of Ogg/WebM to MP4 and vice versa, and to use <source> elements to enumerate the available versions. The H.264 fees seem to kick in only for content that's pay-to-play. ogv.js is very cool though :)

Great, transcoding is the best way if the licenses aren't an issue! :D


[Bug]

Search before asking

[X] I had searched in the issues and found no similar issues.

Apache SkyWalking Component
OAP server (apache/skywalking)
What happened
The configuration document for kafka feeder in the OAP document is outdated, and there are no configuration items for isSharing and consumePartitions in the current version of the configuration

What you expected to happen
OAP documents can be updated to the latest version of Kafka Fetcher
Kafka Fetcher
The Kafka Fetcher pulls messages from the Kafka Broker to learn about what agents have delivered. Check the agent documentation for details on how to enable the Kafka reporter. Typically, tracing segments,  service/instance properties, JVM metrics, and meter system data are supported (depending on the agent implementation). Kafka Fetcher can work with gRPC/HTTP Receivers simultaneously for adopting different transport protocols.
Kafka Fetcher is disabled by default. To enable it, configure it as follows.
Namespace aims to isolate multi OAP clusters when using the same Kafka cluster.
If you set a namespace for Kafka fetcher, the OAP will add a prefix to the topic name. You should also set the namespace in the property named plugin.kafka.namespace in agent.config.
kafka-fetcher:
  selector: ${SW_KAFKA_FETCHER:default}
  default:
    bootstrapServers: ${SW_KAFKA_FETCHER_SERVERS:localhost:9092}
    namespace: ${SW_NAMESPACE:""}

skywalking-segments, skywalking-metrics, skywalking-profilings, skywalking-managements, skywalking-meters, skywalking-logs
and skywalking-logs-json topics are required by kafka-fetcher.
If they do not exist, Kafka Fetcher will create them by default. Also, you can create them by yourself before the OAP server starts.
When using the OAP server automatic creation mechanism, you could modify the number of partitions and replications of the topics using the following configurations:
kafka-fetcher:
  selector: ${SW_KAFKA_FETCHER:default}
  default:
    bootstrapServers: ${SW_KAFKA_FETCHER_SERVERS:localhost:9092}
    namespace: ${SW_NAMESPACE:""}
    partitions: ${SW_KAFKA_FETCHER_PARTITIONS:3}
    replicationFactor: ${SW_KAFKA_FETCHER_PARTITIONS_FACTOR:2}
    isSharding: ${SW_KAFKA_FETCHER_IS_SHARDING:false}
    consumePartitions: ${SW_KAFKA_FETCHER_CONSUME_PARTITIONS:""}
    consumers: ${SW_KAFKA_FETCHER_CONSUMERS:1}

In the cluster mode, all topics have the same number of partitions. Set "isSharding" to "true" and assign the partitions to consume for the OAP server. Use commas to separate multiple partitions for the OAP server.
The Kafka Fetcher allows you to configure all the Kafka producers listed here in property kafkaConsumerConfig. For example:
kafka-fetcher:
  selector: ${SW_KAFKA_FETCHER:default}
  default:
    bootstrapServers: ${SW_KAFKA_FETCHER_SERVERS:localhost:9092}
    namespace: ${SW_NAMESPACE:""}
    partitions: ${SW_KAFKA_FETCHER_PARTITIONS:3}
    replicationFactor: ${SW_KAFKA_FETCHER_PARTITIONS_FACTOR:2}
    isSharding: ${SW_KAFKA_FETCHER_IS_SHARDING:true}
    consumePartitions: ${SW_KAFKA_FETCHER_CONSUME_PARTITIONS:1,3,5}
    consumers: ${SW_KAFKA_FETCHER_CONSUMERS:1}
    kafkaConsumerConfig:
      enable.auto.commit: true
      ...

When using Kafka MirrorMaker 2.0 to replicate topics between Kafka clusters, you can set the source Kafka Cluster alias (mm2SourceAlias) and separator (mm2SourceSeparator) according to your Kafka MirrorMaker config.
kafka-fetcher:
  selector: ${SW_KAFKA_FETCHER:default}
  default:
    bootstrapServers: ${SW_KAFKA_FETCHER_SERVERS:localhost:9092}
    namespace: ${SW_NAMESPACE:""}
    partitions: ${SW_KAFKA_FETCHER_PARTITIONS:3}
    replicationFactor: ${SW_KAFKA_FETCHER_PARTITIONS_FACTOR:2}
    isSharding: ${SW_KAFKA_FETCHER_IS_SHARDING:true}
    consumePartitions: ${SW_KAFKA_FETCHER_CONSUME_PARTITIONS:1,3,5}
    consumers: ${SW_KAFKA_FETCHER_CONSUMERS:1}
    mm2SourceAlias: ${SW_KAFKA_MM2_SOURCE_ALIAS:""}
    mm2SourceSeparator: ${SW_KAFKA_MM2_SOURCE_SEPARATOR:""}
    kafkaConsumerConfig:
      enable.auto.commit: true
      ...

Other Fetcher Plugins
There are other transporter plugins. You can find these plugins from 3rd party repositories.

Pulsar Fetcher Plugin

RocketMQ Fetcher Plugin

How to reproduce

Anything else
No response
Are you willing to submit a pull request to fix on your own?

[X] Yes I am willing to submit a pull request on my own!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

If this is just a doc issue, please submit a pull request to fix directly.


Added xmllint and fleka8

For some reason the original PR was closed https://github.com/ToyotaResearchInstitute/gauges2/pull/14. It was approved


Connect to chat only after call is connected

What
Fix bug when user joined chat before call and message about join appears first.
I've added call to join chat only after call is connected.
Updated ChatAdapter to wait when user joined thread for operations.
Why

How Tested

Process & policy checklist

[ ] I have updated the project documentation to reflect my changes if necessary.
[ ] I have read the CONTRIBUTING documentation.

Is this a breaking change?

[ ] This change causes current functionality to break.

Closing this as there has been no iterations in a significant amount of time. This bug isn't a regression, and will need design at an architecture level to support this feature.


language bug

if I configure my application in the following way:
in app/Config/App.php
public $defaultLocale = 'ar';
public $negotiateLocale = false;
public $supportedLocales = ['ar'];
public $appTimezone = 'America/Argentina/Salta';
public $charset = 'UTF-8';
in app/Config/Boilerplate.php
public $i18n = 'Spanish';
then I get this error. In my-app/admin/user/profile
Exception
DateTime::__construct(): Failed to parse time string (٢٠٢٠-٠٨-٠٣ ١٧:٠٠:٤٩) at position 0 (�): Unexpected character
https://subefotos.com/ver/?e87dadd2778911569dd6270b16cdedd1o.png

The function humanize() is extends from Time() class, i believe codeigniter 4 related bug. Please summit issue to https://github.com/codeigniter4/CodeIgniter4


Bottom part of the image cropped with UTV007

Hi Guys,
I have discovered weird problem. I am not sure if this is an issue with hyperion or openelec. I hope you can help out.
On Openelec Fushicai/UTV007 grabber has image cropped like this (please note the bottom on the picture):

On raspbmc with modified kernel it is perfectly fine:

Output from taking the screenshot is this:
Openelec:
V4L2 width=720 height=480
V4L2 pixel format=YUYV
V4L2 grabber signal threshold set to: {0,0,0}
V4L2 grabber started
V4L2 grabber stopped
Raspbmc:
V4L2 width=720 height=576
V4L2 pixel format=YUYV
V4L2 grabber signal threshold set to: {0,0,0}
V4L2 grabber started
V4L2 grabber stopped
Please note that the height is different. I guess I don't need to add that --height 576 does not change anything. I would be grateful for any hints.
Jacek

@tociek can you close this pls? thanks


Fix lack-of-aliasing bug in ArithmeticOperation

Fixes #2160

I think it's because the axes are always in order in the battery of tests. I'll modify this to add an out-of-order axes case to the test, where the first axis is moved to the end and is not contiguous with the rest of the axes.

@viathor PTAL. I added a test method to the standard battery of tests we use on gates. I checked that it in fact fails on the old arithmetic gate code.