Additional TeX syntax support and direct web-compatible build outputs

This is a fairly major update adding many additional TeX features in addition to adding two more build targets. The updated README has documentation on all of the new user-facing features, whereas the below list touches on those briefly but adds some notes about the internals.
New TeX features:

Variable subscripts: a_b
Custom symbols used in multiple places, such as greek letter variables (\alpha), and certain operators.
Comparison operator support: \ne, <, \geq, etc...
Functions with custom bases: \sqrt[3]{x}, \log_3{y}

Other improvements:

Build scripts for, node outputs (CJS), web (ESM and browser), and all of the above added to package.json
ES6 module and global variable script utilizing Webpack (both work directly in a web browser)
Consistency improvements to parsing logic, such as with strings and symbols
MathJS updated to v10.6.4, where the typing logic of nodes was improved allowing for more specific types utilized here.
Formatting cleanup to reduce linter errors/warnings

The build improvements in this PR cover a similar set of needs to #10, but using Webpack instead of vite.

I also believe this set of changes should also be compatible with MathJS 11, but not MathJS 12.


clean build fails on latest version of Vite

Steps to reproduce

git clone into a fresh directory
Install dependencies
Run pnpm run build in project root and then in demo/

Error
[vite:compress] setAssetSource cannot be called in transform for caching reasons. Use emitFile with a source, or call setAssetSource in another hook.
file: /home/hydrocodone/Repositories/vite-plugin-compress/demo/public/test.svg
error during build:
Error: setAssetSource cannot be called in transform for caching reasons. Use emitFile with a source, or call setAssetSource in another hook.
    at error (/home/hydrocodone/Repositories/vite-plugin-compress/node_modules/.pnpm/rollup@2.45.2/node_modules/rollup/dist/shared/rollup.js:5305:30)
    at throwPluginError (/home/hydrocodone/Repositories/vite-plugin-compress/node_modules/.pnpm/rollup@2.45.2/node_modules/rollup/dist/shared/rollup.js:18040:12)
    at Object.error (/home/hydrocodone/Repositories/vite-plugin-compress/node_modules/.pnpm/rollup@2.45.2/node_modules/rollup/dist/shared/rollup.js:18666:20)
    at Object.error (/home/hydrocodone/Repositories/vite-plugin-compress/node_modules/.pnpm/rollup@2.45.2/node_modules/rollup/dist/shared/rollup.js:18209:38)
    at Object.setAssetSource (/home/hydrocodone/Repositories/vite-plugin-compress/node_modules/.pnpm/rollup@2.45.2/node_modules/rollup/dist/shared/rollup.js:18228:29)
    at Object.transform (/home/hydrocodone/Repositories/vite-plugin-compress/dist/plugin.js:107:34)
    at async ModuleLoader.addModuleSource (/home/hydrocodone/Repositories/vite-plugin-compress/node_modules/.pnpm/rollup@2.45.2/node_modules/rollup/dist/shared/rollup.js:18403:30)
    at async ModuleLoader.fetchModule (/home/hydrocodone/Repositories/vite-plugin-compress/node_modules/.pnpm/rollup@2.45.2/node_modules/rollup/dist/shared/rollup.js:18459:9)
    at async Promise.all (index 1)
    at async ModuleLoader.fetchStaticDependencies (/home/hydrocodone/Repositories/vite-plugin-compress/node_modules/.pnpm/rollup@2.45.2/node_modules/rollup/dist/shared/rollup.js:18485:34)
 ERROR  Command failed with exit code 1.

This could be caused by recent breaking change in Vite v2.3.0. Will submit PR for review.

Thanks for investigating! Awaiting your PR :)

Any update here? We're having this same issue!

Same issue here :/

It's fixed on the next branch, but the fix depends on #1675, which I haven't had time to write tests for.


inivitation: fix "jumping" of modal

closes https://github.com/inveniosoftware/invenio-app-rdm/issues/1600
needs https://github.com/inveniosoftware/invenio-app-rdm/pull/1605

https://user-images.githubusercontent.com/15194802/169237992-f4a0504e-e2d1-4139-aa0b-cf2750b802a4.mp4

in the video there is still change of the height of the modal visible when you are changing tabs, was this the "jump"?
if you select more members (> 5, 10) will the selected members area have a scroll?

in the video there is still change of the height of the modal visible when you are changing tabs, was this the "jump"? if you select more members (> 5, 10) will the selected members area have a scroll?

The "jump" was referring to the change of height of the selected members when selected a member for the fist time. Now the placeholder for "No people selected" has the same height as the initial selected members area. After discussing about the jump when changing tabs during the stand-up we agreed to also add the "Message" text area to the group tab and disable it, to avoid the jumping.
When we select many members the area will expand, there is no scroll, this behaviour was confirmed in the stand-up.


Lifted tuple accessors

In Database.DSH, there's a GADT TupElem with constructors like Tup3_1, Tup3_2 and so on. Their intent seems to be to help implement lifted tuple accessors, e.g.:
tup3_1 :: (QA a, QA b, QA c) => Q (a, b, c) -> Q a

However, no such functions seem to be exported anywhere in DSH. It would be nice if they were. I could submit a pull request but i'm not sure what module such functions should be in, and given that Tup3_1, Tup3_2 and so on seem to be generated with Template Haskell, maybe you would want to generate tup3_1 and tup3_2 with Template Haskell too.

I just added lifted tuple accessors, generated via TH. This has been on my TODO list for a while, thanks for pushing it to the top.
*Database.Algebra.Dag> import qualified Database.DSH as Q
*Database.Algebra.Dag Q> import Database.DSH.Compiler
*Database.Algebra.Dag Q Database.DSH.Compiler> showComprehensionsQ $ Q.map Q.tup3_2 $ Q.toQ ([(1,2,3)] :: [(Integer, Integer, Integer)])
--------------------------------------------------------------------------------
[ v1.2 | v1 <- sng (( 1, 2, 3 )) ]
--------------------------------------------------------------------------------

Note that you can also use view patterns to match on tuples:
*Database.Algebra.Dag Q Database.DSH.Compiler> :set -XViewPatterns
*Database.Algebra.Dag Q Database.DSH.Compiler> showComprehensionsQ $ Q.map (\(Q.view -> (_, b, _)) -> b) $ Q.toQ ([(1,2,3)] :: [(Integer, Integer, Integer)])
--------------------------------------------------------------------------------
[ v1.2 | v1 <- sng (( 1, 2, 3 )) ]
--------------------------------------------------------------------------------

Great, thanks for the quick response. And I didn't know about the view patterns, I'll look into them. Thanks!

Admittedly, documentation is rather inadequate. The best way is probably to go through the examples in https://github.com/ulricha/dsh-example-queries.


Copyright

I think there should be an option to add copyright info, it'll be useful for commercial apps also open-source apps have copyright too.

By copyright do you mean library licenses or something else. You should be able to make it yourself in the meantime - that's the idea of the library.

A copyright info like © 2017 Github Inc at the bottom of screen

You could easily put it under the app name so it looks like this:

appCardBuilder.addItem(new MaterialAboutTitleItem.Builder()
                .text("Material About Library")
                .desc("© 2017 Daniel Stone")
                .icon(R.mipmap.ic_launcher)
                .build());

Okay then.


Improve Exception to NSError conversion

Support NSErrors coming from Swift/ObjC
Support something similar to @Throws

I am currently using something like this:
extension Error {
  var apiError: ApiError? {
    (self as NSError).userInfo["KotlinException"] as? ApiError
  }
}


dismiss keyboard when scrolling home screen

Task/Issue URL: https://app.asana.com/0/392891325557410/1165288656835281
Tech Design URL:
CC:
Description:
Dismiss keyboard when scrolling home screen and fix a bug with padding calculation.
Steps to test this PR:

Add a bunch of favorites.  Check home screen in different modes dismisses the keyboard when scrolling.

Make sure the last row of the favorites is full - ensure centre layout scrolls to the top correctly.

Internal references:
Software Engineering Expectations
Technical Design Template

thanks @bwaresiak


Carbon Ad CSS Stylesheet regression

Link to the Page
All inner pages
Describe the Issue (screenshots encouraged!)
It looks like in the transition to v2 our ad-specific stylesheets was accidentally removed. This means the ads are reverted to their default styles. We need to investigate and resolve.

Turns out the PurgeCSS plugin was purging the ad styles because they are not used in the app at build time. Rather, the ads are injected with JS, so they don't appear until after this process.


upd_20191009 es

update es language 20191009 [content/data/commands-es.json]

@novia713 thanks for the PR

Deployed to https://new.drupalconsole.com/docs/es/


[FIX] remove extraneous bracket in diffuseDetailMap shader chunk

This fixes shader compilation for materials that use diffuseDetailMap textures
I confirm I have read the contributing guidelines and signed the Contributor License Agreement.

Well spotted! 👍


webapp deployment failure

This is an autogenerated template. Please review and update as needed.
Describe the bug
az webapp up fails to create web app.
Command Name
az webapp up
Errors:
:open_mouth:

Parameter 'resource_group_name' must conform to the following pattern: '^[-\\w\\._\\(\\)]+$'.Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 206, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 326, in execute
    raise ex
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 384, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 375, in _run_job
    cmd_copy.exception_handler(ex)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/appservice/commands.py", line 54, in _polish_bad_errors
    raise ex
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 354, in _run_job
    result = cmd_copy(params)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 145, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/__init__.py", line 451, in default_command_handler
    return op(**command_args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/appservice/custom.py", line 2313, in webapp_up
    _create_new_rg = should_create_new_rg(cmd, default_rg, rg_name, is_linux)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/appservice/_create_util.py", line 282, in should_create_new_rg
    elif (_check_resource_group_exists(cmd, rg_name) and
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/appservice/_create_util.py", line 86, in _check_resource_group_exists
    return rcf.resource_groups.check_existence(rg_name)
  File "/opt/az/lib/python3.6/site-packages/azure/mgmt/resource/resources/v2018_05_01/operations/resource_groups_operations.py",line 61, in check_existence
    'resourceGroupName': self._serialize.url("resource_group_name", resource_group_name, 'str', max_length=90, min_length=1, pattern=r'^[-\w\._\(\)]+$'),
  File "/opt/az/lib/python3.6/site-packages/msrest/serialization.py", line 592, in url
    data = self.validate(data, name, required=True, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/msrest/serialization.py", line 672, in validate
    raise ValidationError(key, name, value)
msrest.exceptions.ValidationError: Parameter 'resource_group_name' must conform to the following pattern: '^[-\\w\\._\\(\\)]+$'.

To Reproduce:
Steps to reproduce the behavior. Note: Command arguments have been redacted.

Fill in additional info here
Run: az webapp up -n {}

Expected Behavior
Expected this command to create a web app as described in the quickstart guide.
Environment Summary
Linux-4.15.0-1041-azure-x86_64-with-debian-stretch-sid
Python 3.6.5
Shell: bash

azure-cli                         2.0.62

Additional Context
Trying to deploy a Python Flask web app from a git repository

Getting the same error following the simple guide: https://docs.microsoft.com/en-us/azure/app-service/containers/quickstart-python

@andreyabramov are you still seeing this issue?

closing since we are not to able to repro this on our end, please open a new issue if you still see this with the latest version of CLI.  Thanks!


Implement code highlighting with sourcegraph/syntaxhighlight lib

Sorry for this PR, it's WIP. I was sure I'm creating it in my fork, not in the upstream.

np ;-)


Xnero: Watch n2coin.io

Xnero requests the watch of the watch_keyword n2coin\.io. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

n2coin\.io has been seen in 1 true positive, 0 false positives, and 0 NAAs.

Approved by Jeff Schaller in Charcoal HQ


i need help

when I try to host server I click start hosting for a few seconds it gives me stop hosting then it gives me start hosting and in the meantime it gives me continuously without stopping launching the headless server (the version is 4.5) what should I do

This is not the repo for the launcher however it can be found here https://github.com/Auties00/reboot_launcher


sensor availability not checked ?

I have a max sensor based on the DSMR voltage sensor.
This is not available when HADailySensor is starting and thus gives an error.
Logger: custom_components.daily.sensor Source: custom_components/daily/sensor.py:156 integration: Daily Sensor ([documentation](https://github.com/jeroenterheerdt/HADailySensor), [issues](https://github.com/jeroenterheerdt/HADailySensor/issues)) First occurred: 16:00:21 (6 occurrences) Last logged: 16:00:27 unable to convert unavailable to float. Please check the source sensor (sensor.electricity_meter_voltage_phase_l1) is available.
Adding an availability template to each daily_sensor would prevent this.

what is an availability template?

It checks for the availability of the sensor, refer https://www.home-assistant.io/integrations/template/
The section on "STATE BASED SENSOR CHANGING THE UNIT OF MEASUREMENT OF ANOTHER SENSOR" is the clearest I suppose.
Also a lot of comments on this concept on Tweakers.net (assume you are dutch) and the HA-forum.
code snippet (spacing and code tags are not really working here for some reason;

sensor:

name: "fan speed percentage"
unique_id: "IDsensor_fan_speed_percentage"
state_class: measurement
unit_of_measurement: "%"
icon: mdi:fan
state: >
{{ state_attr('fan.itho_wifi_nrg_21a4','percentage') }}
availability: >-
{{ states( 'fan.itho_wifi_nrg_21a4' ) not in [ 'unknown', 'unavailable'] }}

Right,but that's of course not how it works in python when writing an integration. But I think I get now what you're after.

Thanks a lot Jeroen, Will test it later on.

Logger: homeassistant
Source: custom_components/daily/sensor.py:155
integration: Daily Sensor (documentation, issues)
First occurred: 20:07:49 (172 occurrences)
Last logged: 20:08:49
Error doing job: Future exception was never retrieved
Traceback (most recent call last):
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/daily/sensor.py", line 72, in
lambda event: self._handle_update(  # pylint: disable=unnecessary-lambda
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/daily/sensor.py", line 102, in _handle_update
the_val = self.convert_to_float(input_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/daily/sensor.py", line 155, in convert_to_float
return float(float_value)
^^^^^^^^^^^^^^^^^^
TypeError: float() argument must be a string or a real number, not 'State'
Mm, not really happy after the restart. You want me to make a new Issue?

Same for me...


⚠ Fetchers has degraded performance

In f98a8c4, Fetchers ($STATUS_URL) experienced degraded performance:

HTTP code: 200
Response time: 496 ms

Resolved: Fetchers performance has improved in eb2b00f.


Inline ignore comments issue

I'm trying to use /* uncss:ignore */, but with nested elements in Sass, the comment gets moved to the top of the parent selector, which does not work. If I manually move it back in the exported css, it works fine.
Another example is when I used nested media queries - the output is this:
/* uncss:ignore */
@media only screen and (max-width: 47.9375rem) {
.page-document.expanded {
padding-left: 3.75rem; } }
@media only screen and (min-width:48rem) {
.page-document.expanded {
padding-left: 8.125rem; } }
@media only screen and (min-width:75.0625rem) {
.page-document.expanded {
padding-left: 9.6875rem; } }
Besides adding the ignored classes in the gulp config or rewriting my selectors in Sass, is there anything that can be done about this to persist the comment line on Sass output?

Does Sass move comments marked as important? e.g:
/**! important comment */
@media only screen and (max-width: 47.9375rem) {
.page-document.expanded {
padding-left: 3.75rem; } }

If so you might be interested in this patch, try replacing the uncss install in the gulp plugin with that one. https://github.com/giakki/uncss/pull/148

I think it still formats it, even if it won't remove it from the file after being minified.

@alanontheweb Is this still an issue for you?

@mikelambert This seems abandoned, can you close?


Qs summary endpage

Closing this PR since it has same with below PR
https://github.com/project-sunbird/sunbird-content-player/pull/271


Right eye, split in half Samsung S4

There is an odd situation on the Samsung S4 where the right eye field is cut in half (or shifted over by half). Is there any workaround to this problem?

That's unusual -- I wonder if we have correct device information for the Samsung S4. What is the value if you enter this in the console: navigator.getVRDisplays().then(function (displays) { console.log(displays[0].dpdb_.deviceParams)  })
You can learn more about using devtools on a remote device here if you're unfamiliar

Also, is this reproduceable on this example?

Okay. So interestingly the example works on his device, but this one does not http://devel.lulzbot.com/vr/taz.
I set up a test page to extract the dpdb and here is what I get. It appears as if dpdb is null:

I found out the correct repo to update is https://github.com/WebVRRocks/webvr-polyfill-dpdb. I'll clone that and see if I can do a push request with corrected settings.

What version of the polyfill are you using? I'm getting weird results on my Nexus 6p as well; I'm guessing it's an outdated version of the polyfill (which could potentially clobber updated dpdb values) -- drop the latest build from http://googlevr.github.io/webvr-polyfill/build/webvr-polyfill.min.js and see if that has better results

Also, in looking through the example, I see it has a button for setting full screen. Is this necessary? I had assumed, perhaps incorrectly, that startPresenting would go into full screen for me.

@jsantell: I'll try the latest build. Thank you.

Since the webvr info sample works, and the Taz page does not work for both the Samsung S4 and Nexus devices (my own as well), it sounds like something with your setting up the VR code. I'd take a look at the webvr.info sample source and make sure you're using it correctly.
I'll merge in the Samsung S4 DPDB values in the next release, although the latest version of the polyfill should pull it in from the URL at runtime getting those values since the webvr-polyfill-dpdb has been updated and serves from Github. Hope this helps!

It would be trivial to modify the code so that it is setup exactly as in the WebVR Example source, so I'll do that tomorrow and see if it helps. But just as an rational for why I am doing it differently, the WebVR demo is resizing the canvas based on window.innerWidth/window.innerHeight. This pattern tends to be used a lot in WebGL code, but it does not allow the canvas to be positioned anywhere other than covering the entire window. Over time I have adopted a different pattern that relies on canvas.clientWidth/canvas.clientHeight instead. This gives the designer the flexibility to place the WebGL canvas anywhere on the page -- the designer can still choose to use CSS to make it cover the window, but by choice, not necessity.
While 99% of the WebVR use cases may indeed be full screen, I could imagine a blog post that would wrap text around a WebGL canvas, but yet allow the user to click a button to view that same scene in VR. It would be nice is that were possible.
Anyhow, the demo I am doing for LulzBots is full screen, so like I said it's no big deal to make work exactly like the WebVR Example, but if that does turn out to be the problem, would you be willing to look into why using clientWidth/clientHeight is causing issues? I could make a simple test case from the WebVR Example to demonstrate the issue.
-- Marcio
-- Marcio

The fullscreen issue is probably related/similar to #138 -- if you had a simple test case for this it'd be even easier to fix and would be appreciated!

That's easy :) I cloned this repo and modified the WebGL Example to use clientWidth and clientHeight, as I am doing in the TAZ demo:
http://www.marciot.com/webvr-polyfill/examples/basic
The source is here:
https://github.com/marciot/webvr-polyfill/tree/master/examples/basic
I don't have access to a Nexus, so if you don't mind running it on yours I would really appreciate it. If it fails, then it will be a simple test case of the issue I am encountering.
My variation works fine on my iPhone 6 as well as an Android Fire tablet I use for testing, which is why I had believed it to be correct until now.  Having converted your example, I saw that it has "renderer.setPixelRatio", which my version does not (if I put it in, the image gets distorted on my test devices). If I were to take a wild guess, I bet maybe it has something to do which why my variant works on my devices but not others. I feel the answer is right under my nose, but I don't know what it is.
-- Marcio

I added another example. This shows the reason I think using canvas.clientWidth and canvas.clientHeight is preferable to window.innerWidth and window.innerHeight. It shows the hypothetical case in which the user may not want the canvas to be fullscreen in regular viewing mode. It mostly works on my iPhone, except that the page margin gets applied when going in VR mode.
http://www.marciot.com/webvr-polyfill/examples/wraparound

Thanks for the steps to reproduce @marciot! I'll check this out more next week

A small update on this. Switching the code over to work exactly like WebVR Example, using window.innerWidth/window.innerHeight has been confirmed to have solved the problem.
I'm still holding out hope on getting the version with canvas.clientWidth/canvas.clientHeight to work though. Will let you know as I learn more.

It looks like my canvas.clientWidth/canvas.clientHeight demo works too. I think the fix was adding "renderer.setPixelRatio", which was missing in my code.


Design Layout Mockups for the App (2 screens)

The app's working, but that's just it. A well-polished UI is a must if you want your user to love your app. Our app needs UI polish and the first step towards it would be preparing mockups.
Below is the current screen shot of our app.

REQUIRED:

Mockups of the above two screens.
Mockups must adhere to material guidelines. (e.g. choice of colours, padding, font sizes, etc.)

Be imaginative, but be grounded... After all it needs to be coded too, right?
Good Luck!

Interested in taking up this issue? Leave a comment expressing your interest along with your @slack id.abnd.
Each issue expects a 24hr turnaround time by default. In case where a student doesn't come back with an update after 24 hours have elapsed, the issue is up for grabs by anyone.

Any  feedback @numerative @piedcipher @s3rius @Deepamgoel @geekykant
Most Welcome..

Great Job @iamhshaikh Looks amazing :sparkles: :+1:

This looks great @iamhshaikh, thanks!
I have a couple of changes based on the roadmap 2.0  changes.

Labels Change Task, shuffle icon, and work session completed are not required.
Instead of displaying number 2, it should display two check marks, e.g. ✔ ✔ .

@numerative can you share some sketches??

Here you go. Just an simple as this.

I have created one

Also just small icon hope you will like it


[vimeo] fix album extraction (issue #15704)

Before submitting a pull request make sure you have:

[x] At least skimmed through adding new extractor tutorial and youtube-dl coding conventions sections
[x] Searched the bugtracker for similar pull requests
[x] Checked the code with flake8

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

[x] I am the original author of this code and I am willing to release it under Unlicense
[x] I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

(Mixed, took some snippets from elsewhere in youtube-dl)
What is the purpose of your pull request?

[x] Bug fix

Description of your pull request and other information
Fixes the vimeo album playlist extraction. I added a test (from the original bug report). So, this resolves #15704.

This fix fails to extract password protected albums!
For example (from test) : https://vimeo.com/album/3253534

As said, that's not what I tried to achieve. Please open a seperate Issue, if you haven't done yet.
Regards

@dstftw Is this ready?
Regards

Does not work for multi page albums.
Does not work https://vimeo.com/album/2632481/sort:plays/format:thumbnail.
Does not work https://vimeo.com/album/2632481/page:2/sort:plays/format:thumbnail.

I resolved the things you requested, could you review again please @dstftw
Regards


archive imported photos

At present we just store the URLs of photos imported from external providers like Flickr. While this saves some disk and bandwidth costs, it means we lose these photos if/when the external providers change the URLs (e.g. in response to changes in privacy settings), resulting in broken image URLs. Most people seem to expect that when they import a photo to iNat, iNat has that photo for ever, and the actual data gets transferred, so to meet that expectation and deal with a number of related bugs (e.g. #379), let's start importing the actual photo data.
There are a couple ways this could go down technically

Ditch FlickrPhoto, PicasaPhoto, and FacebookPhoto and make all imported photos LocalPhotos. We would have to download and convert all existing external photos.
Continue letting the photo subclasses work as they do, using the external URLs, but allow them to have a file attachment like LocalPhoto does, and set that with the original photo data when the photo is imported (probably as a background job). Less refactoring, but we would have to periodically check for broken links and fallback to our own image URLs when that happens. We kind of already do that for Flickr, I guess

Thoughts on implementation, @md5 & @pleary? I guess I lean toward 2.

Ok, we decided we'd add a new photo type called RemotePhoto and adopt something like option 1 above. So now this is looking something like

[ ] add new RemotePhoto subclass of Photo that has the same file attachment as LocalPhoto
[ ] add provider_type and native_original_image_url to the photos table
[ ] create a rake task that goes through every Photo with ObservationPhotos that is not a LocalPhoto and and convert it into a RemotePhoto, setting it's file attachment and native_original_image_url using the original_url
[ ] notify users that we are about to make this change and that anyone with broken images needs to relink their account to inat by date X, otherwise their broken photos will be removed from inat
[ ] On date X, repair all broken photos
[ ] once broken photos are repaired or deleted, run the RemotePhoto conversion task

I ended up ditching RemotePhoto and having these be different subtypes of LocalPhoto. Photo will gain subtype and native_original_image_url. All the subclasses still exist and do everything they used to, but the ObservationsController will turn the (unsaved) subclass instances into LocalPhotos with file = remote url. When saved, the photos will be downloaded and processed the same as an uploaded photo. The subtypes can be helpful when needing to re-sync remote metadata or repairing broken photos.
I made Photo.turn_remote_photo_into_local_photo for the conversion, but no proper rake task right now. And the list of users with broken photos has been generated (ask me if you need it).
Open question - what do we do about photo uniqueness? Right now if someone tries to (re)attach a remote photo to an observation that is used in another observation, a reference is made to the same photo instance - a new instance is not created. I assume we should keep the same practice? If someone tries to add a FlickrPhoto, we should first check for a LocalPhoto of subtype FlickrPhoto with the same native_photo_id, then check for a FlickrPhoto with the same native_photo_id (until we convert them all), and if we find either we'll use that same instance.

I assume we should keep the same practice?

Yes, unless there's a compelling reason not to.

How's this look for a rollout plan:

[ ] merge in this work and deploy so newly imported photos start getting archived (@pleary)
[ ] begin migrating all existing photos without broken links (@pleary)
[ ] create a tool that allows people to fix their existing broken photos (@kueda)
[ ] email all users with broken photos to ask them to use the fixit tool (@kueda)

First two can happen ASAP, dealing with the broken stuff is pending my work on the fixit tool.

looks good to me. I'm working on a few specs and will send a pull request shortly

At this point all photos that can be archived have been. The rest are unavailable for one reason or another. Here's the remainder:
FlickrPhoto => 3978,
PicasaPhoto => 956```

All new photos should be coming in as LocalPhotos (I noticed one exception today with an image that was imported as a TaxonPhoto then associated with an observation. I'll keep an eye out for cases like this). I think we're ready to alert the users of the remaining broken photos

Emailed all the users with remaining broken photos.


Nix-style package management

Adds support for nix-style package management by creating a sandbox in a global location for each package to be installed, this means they can be shared across sandboxes using symlinks.
The idea comes from https://github.com/hvr/hackage-matrix-builder/ which does something similar.
Not sure it's ready for wider consumption just yet but I want to see if it will build on CI.

Have started testing this locally and so far so good. This is game changing!

Might be good to add a doc file about how the shared sandboxes work?

More tests, but :100:

This looks awesome!


Don't misuse lua_next and avoid overflows

Using lua_len instead of lua_rawlen in buffer.c leads to more complicated code, is slower, is inconsistent with luaharfbuzz.c and leads to a security problem: The value returned by lua_len can be manipulated through Lua, so it can be set to a much lower value. Then the following loop writes past the end of the buffer, overwriting arbitrary data.
In multiple positions, the code uses lua_next  like

lua_pushnil(L); int i = 0;
while (lua_next(L, 2) != 0) {
  ... = luaL_checkinteger(L, -1);
  ...[i++] = ...;
  lua_pop(L, 1);
}

There are multiple issues with this: First, lua_next pushes two values (the key and the value), so when only one is pop'ed in the end, the key remains on the stack. Also this code assumes that the valued are returned in the canonical order. While this normally happens, it is not guaranteed: To quote the Lua reference manual (about next, but lua_next is just a C name for next):

The order in which the indices are enumerated is not specified, even for numeric indices. (To traverse a table in numerical order, use a numerical for.)

(It is also slightly slower than iterating over the indices directly.)

@khaledhosny Just checking to see if you have the time/inclination to look at this, since you have spent more time in this codebase recently.
If not, I will try and take a look on the weekend.

@deepakjois I adressed the comments from @khaledhosny.


🐛[BUG]: Installation Issues

Version
0.1.0
On which installation method(s) does this occur?
No response
Describe the issue
Installation following the instruction but get errors: error: metadata-generation-failed. Tried both python 3.12 and 3.11.

HI @fengcong1992
Can you use one of the latest version of earth2studio and elaborate what type of install procedure you are using? (docker or conda etc?) Thanks!
Please make sure you pip is up to date: pip install --upgrade pip


[Feature] Metadata consistency and persistent governance

Search before asking

[X] I had searched in the issues and found no similar issues.

Feature Request
As we know, metadata is stored in the memory of the EventMesh. When a large-scale cluster encounters jitters, both consistency and reliability are at risk. Here, we hope to propose a metadata management system based on raft or gossip protocol. The dledger, etcd, or jgroups component can be introduced. Welcome to sign up and get the task:-)
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

ETCD is a good choice.


Update from <IP_ADDRESS> showed error window

Microsoft PowerToys version
<IP_ADDRESS>
Running as admin

[X] Yes

Area(s) with issue?
Installer
Steps to reproduce
The last update failed and asked me to open a bug here with the log file
2022-02-11.txt
✔ Expected Behavior
No response
❌ Actual Behavior
No response
Other Software
Version: <IP_ADDRESS>
OS Version: Microsoft Windows NT 10.0.22000.0
IntPtr Length: 8
x64: True
Date: 2/11/2022 9:08:54 AM
Exception:
System.IO.IOException: The process cannot access the file 'C:\Users\sinan\AppData\Local\Microsoft\PowerToys\PowerToys Run\Cache\Image.cache' because it is being used by another process.
at System.IO.FileStream.ValidateFileHandle(SafeFileHandle fileHandle)
at System.IO.FileStream.CreateFileOpenHandle(FileMode mode, FileShare share, FileOptions options)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
at System.IO.FileStream..ctor(String path, FileMode mode)
at Wox.Infrastructure.Storage.BinaryStorage1.Save(T data)    at Wox.Infrastructure.Storage.BinaryStorage1.TryLoad(T defaultData)
at Wox.Infrastructure.Image.ImageLoader.Initialize(Theme theme)
at PowerLauncher.App.<>c__DisplayClass14_0.b__0()
at Wox.Infrastructure.Stopwatch.Normal(String message, Action action)
at PowerLauncher.App.OnStartup(Object sender, StartupEventArgs e)
at System.Windows.Application.OnStartup(StartupEventArgs e)
at System.Windows.Application.<.ctor>b__1_0(Object unused)
at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)

Duplicate of #15891

Issue is a duplicate
Thank you for your issue! We've linked your report against another issue to centralize the thread.
Thanks for helping us to make PowerToys a better piece of software.


Minor consistency issue with error variables

@wade-welles I reject the change because it breaks build and backward compatibility. The errors are part of library interface and used by external code to check error cases.


Pretrained ckpt

Hi @wwsource,
Thank you for your great work. I want to run the demo with your provided ckpt. However, I cannot download it through the Baidu link. Could you instead upload your pretrained ckpt to GoogleDrive/HuggingFace?
Thank you!

Thank you for noticing our work!
The trained weights are uploaded to Huggingface. You can download it through the link https://huggingface.co/wwcreator/SceneTracker.
I hope it can be helpful for your work~

Great! Thank you.


Using an alias with an absolute path and outside of the root directory

Is your feature request related to a problem? Please describe.
Currently, adding an alias in vite.config.js with an absolute path or a path that is outside of the root directory isn't working.
Configuration example :
const path = require('path')

module.exports = {
  root: path.join(__dirname, 'src'), // E:\vite\src
  alias: {
    __APP_COMPONENTS__: path.join(__dirname, 'components') // An absolute path and outside of the root directory : E:\vite\components
  }
}

Whereas with Webpack's aliases, it works.
Here a repo that illustrates the case with a vite project that doesn't work and an equivalent working webpack project : https://github.com/clemvnt/vite-alias-path-issue
Is that on purpose or can we consider this possible in the future ?

Vite alias is only for aliasing of package names - it doesn't work like webpack alias.
What you want to do is possible by using resolvers instead - example: https://github.com/vuejs/vitepress/blob/master/src/utils/pathResolver.ts#L18
The reason you need to also provide a "fileToRequest" function is so that HMR can work.

The link https://github.com/vuejs/vitepress/blob/master/src/utils/pathResolver.ts#L18 is broken. Could you provide another example?


Don't display spinner when loading state/locals, instead of that grey out

hey @yann300  --  👋  hi there.. kevin from gitcoin.  im considering putting a bounty on this issue to help you all get some fresh interest on them.  any objection?  are there specific github issues that would make good test bounties?  ill bring the budget, you just tell me what you want.

hello!
That's definitely a good issue for that.
There might be other issues that could fit, I'll take a look.
Thanks!

wow, this would be amazing actually.
I'm thinking about our workshops in taiwan and the many learners we have.
They are not beginners but already have quite some background and would like to practice remote work.
This sounds sweet in that context. Our workshop (including teaching them more about ethereum) will happen in january :-)

Nice, would love to help if anything is needed here...

here ya go.. hopefully this adds some value.  ☝

@owocki @yann300 I came across this issue via Gitcoin, could you please provide a screenshot of the issue and elaborate or sketch what's desired please?

hello @ltfschoen
Basically this is where the code has be to changed:
https://github.com/ethereum/remix/blob/master/remix-debugger/src/ui/SolidityState.js#L59
Here is some information on how to setup a dev env: https://github.com/ethereum/remix#contributing . feel free to ask question in gitter : https://gitter.im/ethereum/remix-dev

@yann300 I followed the Remix instructions to setup a dev environment as follows:
git clone https://github.com/ethereum/browser-solidity; cd browser-solidity/; npm install; npm start;
open -a "Google Chrome" http://<IP_ADDRESS>:8080/

I then performed the following actions within the Remix IDE:

In the "Run" tab I selected the "JavaScript VM" environment and then clicked "Create" to deploy the Ballot.sol contract. I then click the "Debug" button in the bottom VM Console window.
I then went to the "Debugger" tab and clicked the Play button to "start debugging"
I then clicked the Step Over button in the dropdown panel in order to "Step over forward" within the function Ballot. Each time I clicked Step Over it caused the Spinner icon (Font Awesome Refresh icon) appeared momentarily within both the "Solidity Locals" and "Solidity State" sections of the "Debugger" tab (i.e. document.querySelector('.dropdownpanel .fa-refresh').style.display = 'inline-block';), and when they disappeared the current values of respective local variables and state variables were shown. The Spinner icon appears even when I Step Over the loading/assigning a local variable i.e. proposals.length = _numProposals;, but I do not see anything "greyed out" (which is the title of this Issue 612).

I still do not fully understand what the objective is for this Issue. The title of the issue is "Don't display spinner when loading state/locals, instead of that grey out". By this do you mean that you want to prevent the spinner from appearing in both the "Solidity Locals" and "Solidity State" sections of the dropdown panel whenever we Step Over or Step Into code that involves "loading" state/locals (i.e. such as copying local variables to state variables in storage, assigning a pointer from a state variable in storage to a local variable also in storage, and calling functions that hand over a reference to a state variable)
References:

Remix Repo Contribution Guidelines https://github.com/ethereum/remix#contributing
Remix Browser Code https://github.com/ethereum/browser-solidity
Solidity Data Location http://solidity.readthedocs.io/en/develop/types.html#data-location

Yes, instead of that we should grey out the content of "Solidity Locals" and "Solidity State" in order to warn that the current is not up to date and is going to be updated.

@yann300 How would this change improve the developer experience?

by doing this change, the height of these panels will be a bit more consistent.

still working on sourcing someone to work on this.  if someone on the thread is working on the issue... mind claiming it on gitcoin?)

Hi @owocki,
Came across gitcoin bounty, looks interesting. Would like to claim this one. I've made some changes to grey out content while its update state. Please check below:

Does this look good? Let me know if chairperson value needs to be grayed out as well.

defer to @yann300 on the above questions.  they're the jefe of this repo :)  im just here to pay the bounty.

nice work @ravirdv! glad to see someone pick up the task. have you created a PR?

I've created PR for this issue, please let me know if any changes are required.

Hi @yann300,
Did you get chance to look into this? looking forward to your feedback.
Thanks.


Fix spotless check on windows

Ensures spotless is looking for the correct line endings on windows.

@zml2008 Thanks for the review. I've made some changes for what you've suggested.

Is the regex pattern I got good, or would I need to be more specific?
Doing *.gradle.kts didn't work as build.gradle.kts for example is supposed to be crlf on windows. Since the spongeforge.settings.gradle.kts is created, I removed the manually specified \n and made it add the seperator depending on the platform. The other gradle files are not affected as they're tracked and changed by git.


Adding multiple languages to PgvectorDocumentstore for multilangual keyword search

Is your feature request related to a problem? Please describe.
In our application, we use PgvectorDocumentStore to store a lot of documents. Our documents can be in english, french, etc.
The problem is that we recently use the PgvectorKeywordRetriever to do some hybrid search and we just see that the document store can only have one language (by default english).
Describe the solution you'd like
I suggest that the parameter  language: str become  languages: List[str], which would define all wanted languages to use.
It would necessitate to also modify the PgvectorKeywordRetriever.run to allow a new parameter language. This allows to use a specific keyword_index per query.
Describe alternatives you've considered
An alternative solution would be to have multiple document store, one for each supported language. It could work but create multiple connections at the same time, I think.

@CharlesCousyn I see this feature as a valuable addition and would like to ask a few clarifying questions:
Are we expecting users to know the language of the text at the time it is inserted into the document store? If so, we could set up an index where the language configuration is specified by another column (e.g., language), as demonstrated in the documentation:
CREATE INDEX pgweb_idx ON pgweb USING GIN (to_tsvector(language, body));

However, I assume this might not be the desired outcome, as it can be difficult or even impossible to determine the language of the text before insertion.
In this case, we could create an index for each desired language, allowing the user to specify the language at query time. This could result in a query like:
WHERE to_tsvector('english', content) @@ plainto_tsquery('english', 'the search query string');

This approach works but poses the risk of false matches since there's no guarantee that the language of the content matches the language specified by the user. All text in the content field will be processed as if it's in the user-specified language (e.g., English in the example above), potentially leading to false positive matches.
If needed, users could mitigate this risk by implementing a method to store the language of the content, perhaps in the document's metadata, and then applying a metadata filter in the query pipeline to ensure alignment between the text's actual language and the query language.
I'm happy to open a PR if you agree.

I think it should be expected that the users know the language of the text at the time it is inserted into the document store!
The decision of what language use should be in the hand of the ones who know the text the best, i think.
A PR is a good idea!

Thanks for the follow-up! @CharlesCousyn
I had to put a little bit of thought into this. After further consideration, I'm leaning towards allowing users to create multiple document stores based on each language. This approach offers flexibility for various scenarios:

When the language is known at insertion time:
Users can simply insert the document into the correct language-specific store.

When the language is unknown at insertion time:
Users can employ the DocumentLanguageClassifier as part of their pipeline, as demonstrated in this example: DocumentLanguageClassifier in a Pipeline

If we were to use a single document store, the solutions that come to mind are the ones I mentioned earlier. Particularly the first would make the most sense if the language is known ahead of insertion.
CREATE INDEX keyword_index ON documents USING GIN (to_tsvector(language, content));

Where the field language contains valid configurations. I'm not particularly fond of this approach as it would require a schema change and be more restrictive when the document's language is unknown at insertion time.
The main drawback of using multiple document stores would be the need for separate keyword extractors for each language. However, this seems like a reasonable trade-off for the flexibility and accuracy it provides.
Do you agree with this direction? I'm open to further discussion or clarification if needed. Perhaps it would be good to get feedback from the core-team before going further.

:wave:
In general, I tend to agree with the solution proposed by @kanenorman, which is simple and do requires no changes (I think).
It seems also consistent with the Haystack idea of Document Store, where each corresponds to a collection.


"applicationinsights 3.1.0" is not compatible with the "@azure/functions 4.5.0"

Hello team,
I receive typescript compilation errors when trying to start my function locally. Please see the screenshots with the examples:

This package is not currently compatible with Azure Functions version 4.5.0. We have upstream support in progress in OpenTelemetry, and will update the thread here #1299.

Thank you for your answer.
This might be out of scope for this conversation, however, can you maybe point me out where to start if I want distributed tracing for my system? I use Azure Function Apps (HTTP, serviceBusTopic, and serviceBusQueue triggers), and Service Bus with both topics and queues respectively. I use Node.js runtime for Function Apps.
Documentation seems quite vague to me. There is a lot of information about monitoring, application insights, opentelemetry, etc., however it is hard to say where to start.
So first I tried to add applicationinsights npm package (basically the initial topic of this issue).
Then I tried @azure/monitor-opentelemetry (from the Docs: "The Distro enables automatic telemetry by including OpenTelemetry instrumentation libraries for collecting traces, metrics, logs, and exceptions, and allows collecting custom telemetry."), but it looks to me that it does nothing since basic metrics and logs are being streamed by Function App even without that package. Moreover, I'm not quite sure this distro is suitable for Function Apps.
Then I found this post on StackOverflow which looks like a semi-answered.
So hoping for some helping hand here.
Thanks!

This will be available in next release, related to trace-id not being shared in Service Bus and Azure Functions is better to track separately in https://github.com/Azure/azure-sdk-for-js repo so Azure SDK folks could be involved as well.


Add more logging to the WaitForGenericK8sObjects meassurment

The logging added lists failed objects whenever it fails.

What type of PR is this?

What this PR does / why we need it:
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:

/assign @tosi3k

/ok-to-test
/lgtm
/approve


Default to Deployment.Size 1

I think this is a very sensible default, but there must be some doc impact here? docs/help/...

done a rebase

need to investigate a test, seems we are doing extra reconciles, don't know the root cause yet.


Is there away where CLIP gives use multiple outputs like Multiple Object Detection

Hi,
Can You Help me, where If we ask the CLIP irrelevant questions with Image and it still tries to fit the score  even if the class is not there, because of the Softmax layer used.
Or
Is there away where CLIP gives use multiple outputs like if we ask for multiple objects in the image and it gives score for each object separately like we do in object detection.

See #82 for related discussions.


Javascript short circuit syntax not respected by syntax highlighting

From @bradbart on April 29, 2016 4:16

VSCode Version: 1.0.0 fa6d0
OS Version: Windows 10

Steps to Reproduce:

Write a function in a javascript function with the following definition:

function test(err) {
    err && return;
    console.log('error');
}

Even though this syntax is valid, VS code underlines the return statement indicating that it expects an expression and that the console.log code is unreachable.
Copied from original issue: Microsoft/vscode#5968

Even though this syntax is valid

No, it isn't. return is a keyword that can only be used in a statement.


Image flickering is taking place while zooming in and out through only OnePlus6T  & Oneplus 7T  mobiles

We are facing image flickering issue while zooming in and out through only Onepus 6T & Onepus 7T devices ,Has anyone encountered such a bug as well what is the fix for this?
we used  "react-responsive-pinch-zoom-pan" this library in our project.
(For refference,I have attached the video ,please check it.)
https://user-images.githubusercontent.com/54093821/131224881-c8949907-f83d-412c-8bdd-bf64784c5d3c.mp4

Can anyone please help ?


You can't install on mac

I noticed that I can't install goosemod on mac because that discorddevelopment folder is missing and that it opens finder but I can't put file name.

You can, system / Discord issue for finding path, not GM.


Does RLock.lock remove key from redis?

When i do RLock.lock, i found that key on the redis will be disappear with given lease time (if not set up then it will be 30 secs).
This is what i saw on redis log, which it using "pexpire" on the key, from what i know, this command will remove the key after the given time.

Should remove the lock only right not the key.
may i know is this true action from RLock.lock or is this a bug
here is my code
RReadWriteLock rwlock = redisson.getReadWriteLock(key);
RLock lock = rwlock.writeLock();
lock.tryLock(0, 5, TimeUnit.SECONDS);

pexpire command invoked by RLock object. Can you provide more details about the issue?

appreciate for your reply.
I think i got wrong idea of lock.
I will close this question


Add advice to run restic check regularly

Followup issue to #1136:

[ ] Add a few prominent advice texts to the restic manual to regularly run restic check
[ ] Add messages to the code that prints something along the lines of "please remember: it is a good idea to run restic check regularly.

Just an idea: some sort of counter somewhere (maybe in cache dir when it'll be available) and print such msg to stderr if 'check' was executed more than X days/backups ago. And reset it after restic check

[ sorry not read 'please-upgrade' page ]. This was already mentioned

Git runs git gc automatically. Does it make sense for you to automatically run (like git does) restic check, let's say every 10 backups. A config switch to opt-out from automatic checks seems useful.

I like the idea of opt-out automatic runs.
However I'd suggest to run the check based on time since last check. This way people who are running hourly backups won't receive a penalty for using their repo more frequently.
Another thing to think about is on what commands the check should be forced. Can it be safe to only force a check on forget/prune?

I'd rather that check is not forced upon me at all! Warn me that check hasn't run in a long time - OK, but I'd rather my backup schedule is predictable and without sudden surprises (oh, I decided to run extra 2 hour check, sorry user).
Having said that, I'm currently implementing backup+prune+check (this last one is new) for some of my repos, but for that to finish in adequate time, I had to revert the patch that disables caching for check command. So far, so good. It reports all the same repo errors I observe without cache, but does it much much faster. 😇
I strongly feel that caching should be implemented first, and also for check command, by default. Together with --force or --no-cache option, when you feel like you want a brute force check the repo itself, bypassing cache. Then all commands would behave the same - by default caching is on - and if you're short on disk space, you can turn that off and trade time for space.

@fd0: The first todo on the list is fixed, can you please add a checkmark so we see the one that's remaining?

done.


Exception while using HoodieStreamer protobuf data from Kafka

I am exploring using Apache Hudi HoodieStreamer to ingest protobuf messages from Kafka into Hudi. Despite a lot of attempts I have hit a roadblock I get an exception while the HoodieStreamer tries make use of the schema from my locally hosted confluent schema registry
Environment Description

Hudi version : 0.15
Spark version : 3.4
Storage (HDFS/S3/GCS..) : Local File System
Running on Docker? (yes/no) : No

I start the HoodieStreamer as follows:
spark-submit \
--packages org.apache.hudi:hudi-utilities-bundle_2.12:0.15.0,org.apache.hudi:hudi-spark3.4-bundle_2.12:0.15.0 \
--jars /home/gaurav/ws/learn/hoodie-delta-streamer/kafka-protobuf-provider-7.6.1.jar \
--driver-memory 8g --executor-memory 8g \
--class org.apache.hudi.utilities.streamer.HoodieStreamer /home/gaurav/ws/learn/hoodie-delta-streamer/hudi-utilities-bundle_2.12-0.15.0.jar \
--props /home/gaurav/ws/learn/hoodie-delta-streamer/kafka/try1/props/kafka-source.properties \
--schemaprovider-class org.apache.hudi.utilities.schema.SchemaRegistryProvider \
--source-class org.apache.hudi.utilities.sources.ProtoKafkaSource \
--table-type COPY_ON_WRITE \
--target-base-path file:\/\/\/home/gaurav/ws/learn/hoodie-delta-streamer/db/try1 \
--target-table vols1 \
--op UPSERT \
--continuous \
--source-limit 4000000 \
--min-sync-interval-seconds 60

The kafka-source.properties passed above is as follows:
hoodie.upsert.shuffle.parallelism=2
hoodie.insert.shuffle.parallelism=2
hoodie.delete.shuffle.parallelism=2
hoodie.bulkinsert.shuffle.parallelism=2
hoodie.datasource.write.recordkey.field=name
hoodie.datasource.write.partitionpath.field=name

hoodie.streamer.schemaprovider.registry.url=http://localhost:8081/subjects/vols1-value/versions/latest
hoodie.streamer.schemaprovider.registry.schemaconverter=org.apache.hudi.utilities.schema.converter.ProtoSchemaToAvroSchemaConverter
hoodie.streamer.source.kafka.proto.value.deserializer.class=io.confluent.kafka.serializers.protobuf.KafkaProtobufDeserializer

hoodie.streamer.source.kafka.topic=vols1

bootstrap.servers=localhost:9092
auto.offset.reset=earliest
schema.registry.url=http://localhost:8081

HoodieStreamer fetches the schema from the schema registry but fails to parse due to a NoSuchMethodException on the ProtoSchemaToAvroSchemaConverter.() method. Here is the relevant exception stack trace:
Caused by: org.apache.hudi.utilities.exception.HoodieSchemaFetchException: Error reading source schema from registry. Please check hoodie.streamer.schemaprovider.registry.url is configured correctly. Truncated URL: http://loc...ons/latest

Caused by: org.apache.hudi.internal.schema.HoodieSchemaException: Failed to parse schema from registry: syntax = "proto3";
package com.gaurav.data.vol;

Caused by: org.apache.hudi.internal.schema.HoodieSchemaException: Failed to parse schema from registry: syntax = "proto3";
package com.gaurav.data.vol;
import "google/protobuf/timestamp.proto";

message VolSurface {
 optional string name = 1;
 repeated .google.protobuf.Timestamp expiry = 2;
 repeated double atmVol = 3;
 repeated double skew = 4;
}

Caused by: org.apache.hudi.exception.HoodieException: Could not load class org.apache.hudi.utilities.schema.converter.ProtoSchemaToAvroSchemaConverter
 at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:64)
 at org.apache.hudi.utilities.schema.SchemaRegistryProvider.parseSchemaFromRegistry(SchemaRegistryProvider.java:107)
 ... 11 more
Caused by: java.lang.InstantiationException: org.apache.hudi.utilities.schema.converter.ProtoSchemaToAvroSchemaConverter
 at java.base/java.lang.Class.newInstance(Class.java:571)
 at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:62)
 ... 12 more
Caused by: java.lang.NoSuchMethodException: org.apache.hudi.utilities.schema.converter.ProtoSchemaToAvroSchemaConverter.<init>()
 at java.base/java.lang.Class.getConstructor0(Class.java:3349)
 at java.base/java.lang.Class.newInstance(Class.java:556)
 ... 13 more

I hope that the use the below line in the properties file is correct . My understanding is that the converter is needed because Hudi needs to convert protobuf message into Avro
hoodie.streamer.schemaprovider.registry.schemaconverter=org.apache.hudi.utilities.schema.converter.ProtoSchemaToAvroSchemaConverter
The NoSuchMethodException on the ProtoSchemaToAvroSchemaConverter.() method (as opposed to a NoClassDefFoundError) seems to indicate that it is failing to find a no-arg constructor and looking at the code downloaded from github that is true, ie there isnt any no-arg constructor for the class ProtoSchemaToAvroSchemaConverter . Unless this exposing a bug, I think that there is something basic I am missing and I will be really grateful if you can point me in the right direction.

Please let me know if I am missing any details in describing the issue or if there is something basic that I am missing here.

@the-other-tim-brown any insights here?

Thank you for providing the attention.
The addition of ProtoKafkaSource is mentioned in the release notes below:
https://hudi.apache.org/releases/release-0.13.0/
Proto Kafka Source
Deltastreamer already supports exactly-once ingestion of new events from Kafka using JSON and Avro formats. ProtoKafkaSource extends this support to Protobuf class-based schemas as well. With just one additional config, one can easily set up this source. Check out the docs for more details.
“

There are some other changes that I need to push for handling schema references within proto that requires a large refactor to the schema registry based provider. I can take a look at that next week when I am back.
We'll also want to include this patch for making sure evolution within a single batch is handled properly: https://github.com/apache/hudi/pull/11373

Thanks @the-other-tim-brown ! Looking forward to it!!
Is my overall usage as described (arguments and properties used)   of the HoodieStreamer for consuming proto from Kafka and confluence schema registry correct or are/will there be changes?
Is the idea here to convert protobuf into avro. Is Avro the format used internally in Hudi for persistence? If yes, does that mean that the original protobuf content will be lost? Is there/ would there be a way to persist the original protobuf (so that it be queried back) along with some extracted fields as separate columns?
Excited about this usage pattern, and sorry if I am asking too many questions.

Lets say I can do without using schema registry and am happy to use a hardcoded schema. Is there a way to consume proto from kafka using that?
I noticed a property called hoodie.streamer.schemaprovider.proto.class.name as described here: https://hudi.apache.org/docs/configurations/#Kafka-Source-Configs
and used it as following in the properties file
hoodie.streamer.schemaprovider.proto.class.name=com.gaurav.data.vol.Volsurf.VolSurface
(this is my protoc compiled class)
I tried using it and also omitted specifying any schema registry properties (with the intention of skipping the usage of schema registry), but looks like properties like schema.registry.url and hoodie.streamer.schemaprovider.registry.url are mandatory.
Without these I get exceptions like
Caused by: io.confluent.common.config.ConfigException: Missing required configuration "schema.registry.url" which has no default value.
at io.confluent.common.config.ConfigDef.parse(ConfigDef.java:251)
at io.confluent.common.config.AbstractConfig.(AbstractConfig.java:78)
at io.confluent.kafka.serializers.AbstractKafkaSchemaSerDeConfig.(AbstractKafkaSchemaSerDeConfig.java:159)
at io.confluent.kafka.serializers.protobuf.KafkaProtobufDeserializerConfig.(KafkaProtobufDeserializerConfig.java:63)
at io.confluent.kafka.serializers.protobuf.KafkaProtobufDeserializer.configure(KafkaProtobufDeserializer.java:55)
at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:717)
... 17 more
and
Caused by: org.apache.hudi.exception.HoodieNotSupportedException: Required property hoodie.streamer.schemaprovider.registry.url is missing
at org.apache.hudi.common.util.ConfigUtils.lambda$checkRequiredConfigProperties$2(ConfigUtils.java:275)
at java.base/java.util.Collections$SingletonList.forEach(Collections.java:4856)
at org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties(ConfigUtils.java:273)
at org.apache.hudi.utilities.schema.SchemaRegistryProvider.(SchemaRegistryProvider.java:168)
If I put those properties back and also hoodie.streamer.schemaprovider.registry.schemaconverter=org.apache.hudi.utilities.schema.converter.ProtoSchemaToAvroSchemaConverter (without that, it tries to use the schema as an avro schema), I get the original error reported in this issue.
I will appreciate if you can provide guidance on whether it is possible to consume protos from kafka without using schema registry as that will also work for me for now.

@gauravg1977 I have put up the changes for the proto schema registry improvements here: https://github.com/apache/hudi/pull/11660
You can use the class base schema provider here instead of the registry. You'll need to set this as your schema provider instead of the Registry base provider


feat: Deprioritize ops methods in completion

Fixes https://github.com/rust-analyzer/rust-analyzer/issues/10593

This currently slows down completions a lot because the lang_item queries aren't cheap and I think the cached values get lost pretty fast, at least the integrated completion benchmark is hitting it twice, even though the second time should be cached.
So I guess we want a better query for fetching the ops traits?

Tried wrapping the lang item queries in an extra query which reduces the performance issues a bit, but still adds 7ms in the case where the query doesnt get reexecuted due to validation checking I suppose(hprof adds a 7ms ??? span). Any ideas on how to do this in a different way maybe?

I think you could just look for the lang attribute on the trait, instead of collecting all the lang items and checking whether it's one of them?

bors r+

hmm this doesn't seem to work in vscode for me, maybe i missed something


DEVTOOLING-180: Refactored response management library

@BrianMoyles Make sure you resolve the merge conflicts and once they are resolved it can be merged


feat: add organization_id to merchant account

Type of Change

[x] New feature

Description

Add field organization_id in merchant account to support for marketplace merchants. Using this an admin user can create many merchant accounts and users under the same organization.
Additional Changes

[x] This PR modifies the API contract
[x] This PR modifies the database schema
[ ] This PR modifies application configuration/environment variables

Motivation and Context

How did you test it?

Create a merchant account with organization id field and check the response to verify that organization_id is populated.

Checklist

[x] I formatted the code cargo +nightly fmt --all
[x] I addressed lints thrown by cargo clippy
[x] I reviewed submitted code

isn't org-id mandatory? Wont all merchants have a default/generated org?

isn't org-id mandatory? Wont all merchants have a default/generated org?
Yes, every merchant will be associated with a org id hencefortth, but for the old application to work with new database changes, it is made as an optional field.


Improve Runtime Performance ?

Ok first of all im new in Golang , And i know i should not ask these kind of questions here , sorry
main question is : is any imporoving performance would happen in future ?
i just realise that Golang is slower then Rust ,  seriously just break my heart ... :(
and i know go and rust is working differently in many things like memory management and stuff like that and they are different type of language...  So im wondering should we expecting major performance improvement at least in version 2 ?
i also know golang compiling code so fast , but its not so big deal, a lot of people dont really care about it , so can we sacrifice compiling speed for runtime speed ?
i dont know any other community for asking these questions ... Thanks for amazing Golang anyway ...

i dont know any other community for asking these questions .

Here is a list of places where you can ask this question.

Stack Overflow with questions tagged "go"

The Go Forum, a web-based forum

Gophers Slack, use the invite app for access. The #general channel is a good starting point.

Go Community on Hashnode with questions and posts tagged with "go"

The golang-nuts mailing list

IRC channel #go-nuts on Freenode

Thanks. I will close this now as this is not a PR.


Document utils::is_http_url

Coverage remained the same at 39.909% when pulling ab0f8b595c99c3ace817a18d3e6ff61e89b257bb on rnestler:document-utils into a9a0b01d6ad3a64472b96a990ca36c66ea1085da on newsboat:master.

Cool, thanks! I'd accept docs for any other function too :)


Fix restart bwc

Description
Full restart bwc only restarts single node.
Related Issues
N/A
Check List

[x] New functionality includes testing.
[x] New functionality has been documented.
[x] API changes companion pull request created.
[x] Commits are signed per the DCO using --signoff.
[x] Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@finnegancarroll is this still required since the version bump?

Yea this is a fix for the full cluster restart bwc. Behavior should be all nodes restart but currently only one is upgraded. Introduced back here: https://github.com/opensearch-project/asynchronous-search/pull/575/files#diff-49a96e7eea8a94af862798a45174e6ac43eb4f8b4bd40759b5da63ba31ec3ef7L396


Add temp URL to slugs in sidebar details link.

This was already done for the 'about' panel; Added it to sidebar details also. This is a temp fix to keep beta users from experiencing errors when linking to pages outside of Destinations

:+1:


Upgrade RediSearch

Version 2 of RediSearch has been released. This introduces breaking changes to the API. Unfortunately upgrading to the new version is going to be invasive. The following changes are required:

Update the RediSearch data loaders and the schema
Update the RediSearch Python module used by all the microservices
Update our custom AsyncIO wrapper (and submit as a pull request so we don't have to maintain it moving forward!)

Done in commit c0cf8b10414760ee64c1123164e52d714fa08d77.


Retrain from scratch

adresses #200
This turned out to be a bit more involved, so I made a PR.
Changes

SNL now also has a retrain_from_scratch_each_round attribute
all methods (SRE, SNL, SNPE) have this argument now, so I moved it to NeuralInference.
fixed bugs related to the argument in SRE and SNPE
however, the performance of SNPE is really poor when using retrain_from_scratch_each_round=True. Hence, I give a warning if the user sets it to True for SNPE

This does not have to go into master urgently, so it's probably best if I wait for Alvaro's PR and then rebase it (there will be merge conflicts in SRE).

Closing PR because rebase is turning into a nightmare. Will start from scratch.


Update ext/livereload.js to version 2.2.3 (or 2.3.0)

What version are you using? (Don't say latest. Be specific.)
0.6.3
Expected result
ext/livereload.js to be either 2.2.3 or (today to be released) 2.3.0
Actual result
ext/livereload.js is currently version 2.2.2
Steps to reproduce issue
Why is this important?
The 2.2.3 version (also on NPM) fixes the long overdue ProtocolError is not defined error.
This means an update can replace #116 so I'll close that.

@smhg This is awesome. Thank you. Somehow I've not been seeing notifications on these issues and I'm not sure why. I'll make sure this is in the next release which I'm trying to prep. Could you possibly help me out by just making a PR that includes the new version of the external file?

PR #118 should be it. Let me know if anything is unclear.

Awesome!

@napcs it would be awesome if you could release a new version with the updated client.

@smhg see other message - 0.7.0 is released to npm.

Thx a lot! I forgot I already pinged you in another issue. My appologies.


Defective Docker Issue

Getting this message when trying to build the docker file:
Successfully built df2db614b282
Successfully tagged ibmosquito/web-hello-c:1.0.0
docker rm -f web-hello-c 2>/dev/null || :
docker run -it --name web-hello-c -p 8000:8000 --volume pwd:/outside ibmosquito/web-hello-c:1.0.0 /bin/bash
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/bin/bash": stat /bin/bash: no such file or directory: unknown.
ERRO[0000] error waiting for container: context canceled
make: *** [Makefile:14: dev] Error 127

@t-fine and @johnwalicki I'm not sure if there is an error in the Dockerfile or it is some peculiarity of performing a Docker build within WSL 2 due to limited OCI support, but running make build is failing with the above error for @julianchen215  in WSL 2.
NOTE: We followed @TheMosquito instructions for configuring systemd in WSL 2, and then successfully installed the agent and registered it with the LF Edge lab exchange instance.  Then after cloning this repo, we ran make dev and it failed at the very end of building the Docker image.

The docker build was  successful.
dev: build stop
	docker run -it --name $(NAME) -p $(PORT):$(PORT) --volume `pwd`:/outside $(DOCKER_HUB_ID)/$(NAME):$(VERSION) /bin/bash

fails on the interactive docker run /bin/bash because this is a
FROM scratch

container (see the Dockerfile that does not have bash.
We should delete the make dev  rule. It will never work.
To test, you need to docker log -f the running detached container.

To be clear, The bug is not in the Dockerfile.  The bug is in the Makefile.

If you want to keep the dev: stanza, do this:
Change the -it to -d
docker run -d --name $(NAME) ...
docker logs -f $(NAME)

Mostly fixed by #11
Although the make exec rule should be removed because it tries to run /bin/bash on a scratch container (that does not include bash.
$ make -n exec
docker exec -it web-hello-c /bin/bash


add tests for formatters that use rules (currently failing)

Issue #, if available: https://github.com/aws-cloudformation/cfn-python-lint/issues/1930
Description of changes: This adds some tests for formatters that make use of the rules argument.
They are currently failing because I believe the current code to be incorrect. It only uses the rules from the last template that is checked.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

replacing with #2330 and includes fixes


(feat)First step towards full xml-configuration support

I confirm that this contribution is made under the terms of the license found in the root directory of this repository's source tree and that I have the authority necessary to make this contribution on behalf of its copyright owner.
Why
Adding support for tags far down in the tree, ref #895, results in a lot of code, and with our current approach of using jinja-templates requires a lot of effort each time.
This PR is in preparation to allow for supporting any services.xml configuration from pyvespa, with the approach used by FastHTML as inspiration.
How
By adopting the approach in this PR, we only need to provide a list of valid xml-tags for each configuration file, and all the classes will be auto-generated.
E.g. to express the following services.xml:
<?xml version="1.0" encoding="utf-8" ?>
<services version='1.0' xmlns:deploy="vespa" xmlns:preprocess="properties">

  <container id='default' version='1.0'>
    <nodes count='1'/>
    <component id='ai.vespa.examples.Centroids' bundle='billion-scale-image-search'/>
    <component id='ai.vespa.examples.DimensionReducer' bundle='billion-scale-image-search'/>
    <component id="ai.vespa.examples.BPETokenizer" bundle='billion-scale-image-search'>
      <config name="ai.vespa.examples.bpe-tokenizer">
        <contextlength>77</contextlength>
        <vocabulary>files/bpe_simple_vocab_16e6.txt.gz</vocabulary>
      </config>
    </component>
    <model-evaluation>
      <onnx>
        <models>
          <model name="text_transformer">
            <intraop-threads>1</intraop-threads>
          </model>
          <model name="vespa_innerproduct_ranker">
            <intraop-threads>1</intraop-threads>
          </model>
        </models>
      </onnx>
    </model-evaluation>
    <search>
      <chain id='default' inherits='vespa'>
        <searcher id='ai.vespa.examples.searcher.DeDupingSearcher' bundle='billion-scale-image-search'/>
        <searcher id='ai.vespa.examples.searcher.RankingSearcher' bundle='billion-scale-image-search'/>
        <searcher id="ai.vespa.examples.searcher.CLIPEmbeddingSearcher" bundle="billion-scale-image-search"/>
        <searcher id='ai.vespa.examples.searcher.SPANNSearcher' bundle='billion-scale-image-search'/>
      </chain>
    </search>
    <document-api/>
    <document-processing>
      <chain id='neighbor-assigner' inherits='indexing'>
        <documentprocessor id='ai.vespa.examples.docproc.DimensionReductionDocProc'
                           bundle='billion-scale-image-search'/>
        <documentprocessor id='ai.vespa.examples.docproc.AssignCentroidsDocProc'
                           bundle='billion-scale-image-search'/>
      </chain>
    </document-processing>
  </container>

  <content id='graph' version='1.0'>
    <min-redundancy>1</min-redundancy>
    <documents>
      <document mode='index' type='centroid'/>
      <document-processing cluster='default' chain='neighbor-assigner'/>
    </documents>
    <nodes count='1'/>
    <engine>
      <proton>
        <tuning>
          <searchnode>
            <feeding>
              <concurrency>1.0</concurrency>
            </feeding>
          </searchnode>
        </tuning>
      </proton>
    </engine>
  </content>

  <content id='if' version='1.0'>
    <min-redundancy>1</min-redundancy>
    <documents>
      <document mode='index' type='image'/>
      <document-processing cluster='default' chain='neighbor-assigner'/>
    </documents>
    <nodes count='1'/>
    <engine>
      <proton>
        <tuning>
          <searchnode>
            <requestthreads>
              <persearch>2</persearch>
            </requestthreads>
            <feeding>
              <concurrency>1.0</concurrency>
            </feeding>
            <summary>
              <io>
                <read>directio</read>
              </io>
              <store>
                <cache>
                  <maxsize-percent>5</maxsize-percent>
                  <compression>
                    <type>lz4</type>
                  </compression>
                </cache>
                <logstore>
                  <chunk>
                    <maxsize>16384</maxsize>
                    <compression>
                      <type>zstd</type>
                      <level>3</level>
                    </compression>
                  </chunk>
                </logstore>
              </store>
            </summary>
          </searchnode>
        </tuning>
      </proton>
    </engine>
  </content>
</services>

We can do this from python with:
generated_services = services(
            container(id="default", version="1.0")(
                nodes(count="1"),
                component(
                    id="ai.vespa.examples.Centroids",
                    bundle="billion-scale-image-search",
                ),
                component(
                    id="ai.vespa.examples.DimensionReducer",
                    bundle="billion-scale-image-search",
                ),
                component(
                    id="ai.vespa.examples.BPETokenizer",
                    bundle="billion-scale-image-search",
                )(
                    config(name="ai.vespa.examples.bpe-tokenizer")(
                        vt(
                            "contextlength", "77"
                        ),  # using vt as this is not a predefined tag
                        vt(
                            "vocabulary", "files/bpe_simple_vocab_16e6.txt.gz"
                        ),  # using vt as this is not a predefined tag
                    ),
                ),
                model_evaluation(
                    onnx(
                        models(
                            model(name="text_transformer")(intraop_threads("1")),
                            model(name="vespa_innerproduct_ranker")(
                                intraop_threads("1")
                            ),
                        ),
                    ),
                ),
                search(
                    chain(id="default", inherits="vespa")(
                        searcher(
                            id="ai.vespa.examples.searcher.DeDupingSearcher",
                            bundle="billion-scale-image-search",
                        ),
                        searcher(
                            id="ai.vespa.examples.searcher.RankingSearcher",
                            bundle="billion-scale-image-search",
                        ),
                        searcher(
                            id="ai.vespa.examples.searcher.CLIPEmbeddingSearcher",
                            bundle="billion-scale-image-search",
                        ),
                        searcher(
                            id="ai.vespa.examples.searcher.SPANNSearcher",
                            bundle="billion-scale-image-search",
                        ),
                    ),
                ),
                document_api(),
                document_processing(
                    chain(id="neighbor-assigner", inherits="indexing")(
                        documentprocessor(
                            id="ai.vespa.examples.docproc.DimensionReductionDocProc",
                            bundle="billion-scale-image-search",
                        ),
                        documentprocessor(
                            id="ai.vespa.examples.docproc.AssignCentroidsDocProc",
                            bundle="billion-scale-image-search",
                        ),
                    ),
                ),
            ),
            content(id="graph", version="1.0")(
                min_redundancy("1"),
                documents(
                    document(mode="index", type="centroid"),
                    document_processing(cluster="default", chain="neighbor-assigner"),
                ),
                nodes(count="1"),
                engine(
                    proton(
                        tuning(
                            searchnode(
                                feeding(concurrency("1.0")),
                            ),
                        ),
                    ),
                ),
            ),
            content(id="if", version="1.0")(
                min_redundancy("1"),
                documents(
                    document(mode="index", type="image"),
                    document_processing(cluster="default", chain="neighbor-assigner"),
                ),
                nodes(count="1"),
                engine(
                    proton(
                        tuning(
                            searchnode(
                                requestthreads(persearch("2")),
                                feeding(concurrency("1.0")),
                                summary(
                                    io(read("directio")),
                                    store(
                                        cache(
                                            maxsize_percent("5"),
                                            compression(
                                                vt_type("lz4")
                                            ),  # Using vt_type as type is a reserved keyword
                                        ),
                                        logstore(
                                            chunk(
                                                maxsize("16384"),
                                                compression(
                                                    vt_type(
                                                        "zstd"
                                                    ),  # Using vt_type as type is a reserved keyword
                                                    level("3"),
                                                ),
                                            ),
                                        ),
                                    ),
                                ),
                            ),
                        ),
                    ),
                ),
            ),
            version="1.0",
        )

[x] Added unit tests to check for equality between original xml and generated xml.

Notes

A separate PR will need to integrate this functionality while still ensuring compatibility with existing approach.
This approach can also be reused for other xml configuration files.
This PR only contains the foundation for doing this, and we will need to adapt the ApplicationPackage().services_to_text to use this functionality for it to take effect.
An added bonus is that we are now able to validate any of the xml configuration files against the relaxng-schema.
This also forms the foundation to allow us to generate pyvespa code from xml, like it is done here

There are many files in this PR because:

["**/services.xml", "**/validation-overrides.xml", "**/hosts.xml"] from sample-apps repo, to use as test files for the service configuration.
.rnc and .rng files from vespaengine/vespa for schema validation.

Only the 2 python files needs a review.

Thanks. Yes - that is the goal of next iterations.
On Fri, Sep 20, 2024 at 9:27 AM Jo Kristian Bergum @.***>
wrote:

@.**** approved this pull request.
This is a great direction, unlocking all the features of Vespa - I assume
that excisting functionaliy works and that we can add examples of using the
new syntax?
—
Reply to this email directly, view it on GitHub
https://github.com/vespa-engine/pyvespa/pull/915#pullrequestreview-2317479145,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AF3M74H6FQJJOKYGQNRAKY3ZXPE6PAVCNFSM6AAAAABOJGSC6OVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDGMJXGQ3TSMJUGU
.
You are receiving this because you authored the thread.Message ID:
@.***>


Optimise dsl

@hamnis @rossabaker I had to make changes in PrometheusClientMetricsSpec and DropwizardClientMetricsSpec as the tests were failing because the the uris were missing a leading / and were being matched against Root / ....


Image API

Copied from https://github.com/d3/d3-geo/issues/35
There's room for building a higher level module on top of the existing tile code for working with images. Currently each of the examples re-implements image loading logic separately, along the lines of the following (from Raster & Vector II):
var raster = svg.append("g");
...
function zoomed() {
  ...
  var image = raster
      .attr("transform", "scale(" + tiles.scale + ")translate(" + tiles.translate + ")")
    .selectAll("image")
      .data(tiles, function(d) { return d; });

  image.exit()
      .remove();

  image.enter().append("image")
      .attr("xlink:href", function(d) { return "http://" + ["a", "b", "c"][Math.random() * 3 | 0] + ".tile.openstreetmap.org/" + d[2] + "/" + d[0] + "/" + d[1] + ".png"; })
      .attr("width", 1)
      .attr("height", 1)
      .attr("x", function(d) { return d[0]; })
      .attr("y", function(d) { return d[1]; });
}

I wonder if it might make sense to try to encapsulate this logic into an API similar to that of Leaflet. The API could take as input the raster g element and A URL pattern for tiles, and expose methods for setting the scale and translate. Examples of features that could be added into such a module include:

Tile URL template strings like 'http://{s}.tile.osm.org/{z}/{x}/{y}.png'. This would make it easy to use different tile providers based on declarative data, like in this Leaflet Providers Demo.

Caching tiles. As an example, Raster & Vector II has the behavior of re-loading the same tiles again and again, due to the randomness introduced in the URL generation. I think this issue would be solved by deterministic URL generation, leaving the actual caching up to the browser.

Showing larger, blurry tiles in the background as smaller tiles load in the foreground. This would vastly improve the user experience when zooming. The current behavior in the examples is that the larger tiles disappear, and the user is looking at empty white space until the higher zoom level tiles are loaded. This would however introduce complexity to the code. The exiting of tiles would be contingent upon all of their "child" tiles having loaded.

I did some experimenting towards the blurry background tiles concept, and realized that tiles.scale and tiles.translate changes completely based on the zoom level. Having the blurry tiles in the background until all of their quadtree children are finished loading requires that we can compute the scale and translate for those tiles that are no longer returned from the tile layout.
Maybe instead of having tiles.scale and tiles.translate attached to the returned array, the logic for computing those values could go into a function that takes the tile address as input and returns the transform as output on a per-tile basis. This could be something like tile.transformForTile(address), returning a zoom transform.
We'll also need to listen for when tiles load. The following works for this:
image.enter().append("image")
    .attr("xlink:href", function(d) { ... })
    .on("load", function (d){
      // If this tile is no longer included in the output from the layout,
      // and all of its 4 "children" tiles are loaded,
      // Then remove it from the set of displayed tiles.
    });

In addition, the tiles that are displayed will need to be managed. One idea I had for doing this is to use a Set like this:
var tilesSet = d3.set();
tile
  .scale(transform.k)
  .translate([transform.x, transform.y])
  ()
  .forEach(function (d){
    tilesSet.add(JSON.stringify(d));
  });

The tile addresses could be recovered like this:
var tileAddresses = tilesSet.values().map(JSON.parse);

Closing this to inactivity, and also because it feels like a higher-level API might belong better as a separate module.


Witness legacy support

Aim of this PR is to have full support for Witness and Legacy TX in deposit flows
closes: #309
Funding flow bitcoin-spv connections. Proof functions not included (not version dependent):

[x] validateVin
[x] validateVout
[x] determineInputLength
[x] extractScriptSigLen
[x] determineVarIntDataLength
[x] extractOutputAtIndex
[x] determineOutputLength
[x] extractHash

Redemption flow.

[x] extractOutpoint
[x] extractValue
[x] extractHash

@prestwich from what i've seen all bitcoin-spv functions used in tbtc context should support legacy txs as well. Is there something I'm missing?

Ok, rebased this and adjusted for test evolution. Time to merge.


Fixed Test-DbaWindowsLogin so it copes with Computer accounts

Type of Change

[x] Bug fix (non-breaking change, fixes #3792 )
[x] New feature (non-breaking change, adds functionality)
[ ] Breaking change (effects multiple commands or functionality)
[x] Ran manual Pester test and has passed (`.\tests\manual.pester.ps1)
[ ] Adding code coverage to existing functionality
[ ] Pester test is included
[ ] Nunit test is included
[ ] Documentation
[ ] Build system

Purpose
Implements fix for #3792
Lots of apps use the AD computer account to connect (SCCM, CRM, etc) and Test-DbaWindowsLogin doesn't parse them properly
Approach
If the account isn't matched as user, check if it's in computer name format ',*`$' and then lookup as a computer
Screenshots
Before/After:

looks good. Only thing that comes to mind are MSA and gMSAs, which have a similar format (did something on those on spn commands).
Did you try with those too ?
Unfortunately I'm still on an unsupported AD version to test those properly :-(

The only difference in format of the computer account and MSA/gMSA is that before the $ it has the same restrictions of any AD username (which means it can't contain any of these " / \ [ ] : ; | = , + * ? < >). So basically just - or _ in the name.
I don't see how it is handled here being an issue on either. If you wanted to take it as an MSA or gMSA though we would have to look up the property that shows it as one.

No problem then
Il dom 15 lug 2018, 05:13 Shawn Melton<EMAIL_ADDRESS>ha
scritto:

The only difference in format of the computer account and MSA/gMSA is that
before the $ it has the same restrictions of any AD username (which means
it can't contain any of these " / \ [ ] : ; | = , + * ? < >). So
basically just - or _ in the name.
I don't see how it is handled here being an issue on either. If you wanted
to take it as an MSA or gMSA though we would have to look up the property
that shows it as one.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/sqlcollaborative/dbatools/pull/3793#issuecomment-405064248,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAHdB11jCm3aMkA_hqlueLa7r5H7yKAtks5uGrNwgaJpZM4VOgC6
.

cool, so merge? 🎉

Yeah, mine was just a sidenote on MSAs that unfortunately I'm unable to
test. Everything is completely fine
Il dom 15 lug 2018, 17:17 Chrissy LeMaire<EMAIL_ADDRESS>ha
scritto:

cool, so merge? 🎉
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/sqlcollaborative/dbatools/pull/3793#issuecomment-405098594,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAHdB7tm3t4XQYEUP6vfOCEeYfrFiX6aks5uG10MgaJpZM4VOgC6
.

thank you all!


Checksum md5sum mismatch when using Joystick for takeoff and landing

I am having a similar issue to this: #100, my program works when I run it for one quadcopter but as soon as I add more than one or two I get the following error:
[ERROR] [1532039584.691694366]: client wants service /crazyflie0/takeoff to have md5sum d41d8cd98f00b204e9800998ecf8427e, but it has b665b6c83a196e4774268cc26329b159. Dropping connection.
[ERROR] [1532039584.692239858]: client wants service /crazyflie3/takeoff to have md5sum d41d8cd98f00b204e9800998ecf8427e, but it has b665b6c83a196e4774268cc26329b159. Dropping connection.
[ERROR] [1532039584.692772]: bad callback: <bound method Controller._joyChanged of <main.Controller instance at 0x7fa88cca1a70>>
Traceback (most recent call last):
File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/topics.py", line 750, in _invoke_callback
cb(msg)
File "/home/quadcopteruser/crazyflie_ws/src/crazyflie_ros/crazyflie_demo/scripts/controller.py", line 46, in _joyChanged
self._takeoff()
File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 435, in call
return self.call(*args, **kwds)
File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 505, in call
raise ServiceException("unable to connect to service: %s"%e)
I have tried reinstalling the crazyflie_ros package and updating my ROS distribution (Kinetic) but neither have had any effect. I run all my nodes on one linux machine running Ubuntu 16.04 and connect to a windows server to communicate with the vicon system.  Any advice on what may be causing this issue?

Which launch file are you using? (Or provide some steps to reproduce the problem, please).

I am using a custom launch file where I hover more than one quadcopter, they each have their own crazyradio and unique URI. Usually the launch file will run without problem until I press the joystick button for land or takeoff which causes the checksum error. I never get the issue with a single quadcopter. The problem occurs when I have more than one crazyflie node in the launch file and the more crazyflies I have, the more often the issue occurs. However, rolling back to the previous version of the crazyflie_ros package mentioned in "Flying Multiple UAVs Using ROS" fixed my problem.

Hi, I'm facing the same issue!
I'm using ROS on a virtual machine of linux 16.04
I would like to autonomously drive 4 CF's with the help of LPS (6 anchors) in ROS environment, but I got stuck.
I've followed the: https://www.bitcraze.io/getting-started ... yflie-2-0/ to update the firmware of all the 4 crazyflies.
I've followed the: https://www.bitcraze.io/getting-started ... ng-system/ for setting the Loco Positioning System in the right way.
I've launch the: dwm_loc_ekf_hover.launch with a 6 anchors setup in both TWR and TdOA mode, and it works!
In TdOA I've even tryed with 2 CF's and it works!
Now I'm trying to launch the: dwm_loc_ekf_swarm_hover.launch with 4 CF's but I found necessary to add another Crazyradio.
So I' ve put 2 CF's on one Crazyradio and 2 CF's on the other Crazyradio changing the dwm_loc_ekf_swarm_hover.launch with the appropriate URI:
uri0 radio://0/80/2M/E7E7E7E701
uri1 radio://1/70/2M/E7E7E7E702
uri2 radio://1/70/2M/E7E7E7E703
uri3 radio://0/80/2M/E7E7E7E704
Unexpectedly the result was something like this:

Could anyone help me in find what could be the problem?
Thank you very much.
P.S:
I'm pretty sure that the previous URI setup is correct because implementing it in the multi_teleop_xbox360.launch (https://github.com/whoenig/crazyflie_ros) I could manually drive the 4 CF's.

Hi, @cjbanks and @lorenzol92
I'm facing the similar issue. #124
Have you already done with the problem?

@cjbanks @lorenzol92
OK, I've done, by changing the firmware.

@shmpwk could you please tell me which version ?
I tested with all 2019.09 , 2019.01 and 2018.01 versions and still experiencing same issue.

Experiencing Similar issue !
I am using motion Analysis tracking system. successfully  hovered one Crazyflie 2.1 and Also trajectory movements after some changes to code and making it working with motion analysis system.
However, when I am trying to hover two devices. I run into checksum Error ( Attached ScreenShot)
I have tested with firmware versions :

2019.01

2018.01

2019.09

All shows same error messages. ( after pushing take off too many times JUST one quad start hovering)

full detail log : https://pastebin.com/E7atN8pA
my modified multi_hover_vicon.launch : https://pastebin.com/s4vGP0hb
my rqt_graph :

@whoenig  Please give me some suggestions to see whether works.


Remove data and pass only files to session.post

This Pull Request is to fix #16.
In the issue I was wrong, it is not the context manager that is triggering /file_close too early but the 3 Traceback shown there were actually triggered by the POST to /file_write.
The problem is that we are sending the data twice:

First time here with data=b
https://github.com/tomgross/pycloud/blob/1b0d9de6521904dcdc3ca3295f544008065e9e4e/src/pcloud/pcloudfs.py#L51
Second time here where we are passing files and kwargs, that in our case are fd, again data and auth
https://github.com/tomgross/pycloud/blob/1b0d9de6521904dcdc3ca3295f544008065e9e4e/src/pcloud/api.py#L221

The fact that with a simple "HelloWorld" file was working I think it was because of the length of data was still very small but my knowledge of requests is limited, so I'm not 100% sure yet.

Hi @blasterspike Thanks for the detailed analysis and the patch!
I have to dig in myself a little bit because I think something is fundametaly broken.
file_write does not implement the pCloud API. There are no addional parameters see https://docs.pcloud.com/methods/fileops/file_write.html
And why is the data transfered to the filename?

Hi @tomgross,
From this other documentation page
https://docs.pcloud.com/methods/fileops/index.html
you can read that with file_write you can use a

POST with 'multipart/form-data' encoding.  In this case the data should be send in the field named "files" found after any post parameters

When you pass files to requests, it does automatically the 'multipart/form-data' encoding for you:
http://docs.python-requests.org/en/master/user/quickstart/#post-a-multipart-encoded-file
You can search for "Content-Type: multipart/form-data;" in the debug output that I have posted in the issue to see that it is doing it correctly.