Warnings from ::Vips::Image.new_from_array

After upgrade to ruby-vips 2.1.0 I get the following warnings on ubuntu focal with vips 8.9.1:
# test.rb

require "vips"

::Vips::Image.new_from_array [[-1, -1, -1],
                              [-1, 32, -1],
                              [-1, -1, -1]], 24

Output:
(test.rb:40847): VIPS-WARNING **: 11:21:01.732: can't set metadata "scale" on shared image

(test.rb:40847): VIPS-WARNING **: 11:21:01.732: can't set metadata "offset" on shared image

To Reproduce
Use the following bash script on an ubuntu focal machine:
# install libvips if not installed
apt install libvips42

# create Gemfile
echo "source 'https://rubygems.org'
gem 'ruby-vips'" > Gemfile

# install gems
bundle install

# create test file
echo 'require "vips"

::Vips::Image.new_from_array [[-1, -1, -1],
                              [-1, 32, -1],
                              [-1, -1, -1]], 24' > test.rb

# run test file
bundle exec ruby test.rb

Expected behavior
No warnings
Environment

OS: Ubuntu Focal

Version 20.04.2

Vips version: libvips42:amd64/focal 8.9.1-2 uptodate

Gemfile.lock
 GEM
   remote: https://rubygems.org/
   specs:
     ffi (1.15.0)
     ruby-vips (2.1.0)
       ffi (~> 1.12)

 PLATFORMS
   x86_64-linux

 DEPENDENCIES
   ruby-vips

 BUNDLED WITH
    2.2.7

Additional context
I thought it was an issue with the image processing gem so I opened an issue there, too: https://github.com/janko/image_processing/issues/83
I also have some image corruption after the upgrade to ruby-vips 2.1.0, but it only happens on my ubuntu focal server and not locally on my mac os development machine (which has a more recent vips version). Maybe the warnings are the cause.

Hi Markus, thanks for the detailed report.
Yes, this is ruby-vips 2.1 needing a more recent libvips.
For now, I'd go back to an earlier ruby-vips. I'll see if I can update ruby-vips 2.1 to work with libvips 8.9.1.
Sorry about this :(

Thanks for the information. Just a note: It happens with the latest ubuntu LTS, so it probably happens for others, too.
If there is no (easy) fix I think ruby-vips should raise an error if an old version of vips is detected, so at least users know to downgrade ruby-vips or upgrade vips (if possible).
Is it mutate that does not work correctly so even other things than ::Vips::Image.new_from_array are broken, too?

libvips 8.9.0 and 8.9.1 had a horrible misfeature that blocked metadata changes unless refcount was 1.
This was a really bad idea and 8.9.2 (I think) changed it. Anyway, it prevents ruby-vips from modifying metadata, even when using mutate. I might be able to work around it, I'm not sure.

Alright, thanks. It works correctly with ruby-vips 2.0.17 and vips 8.9.1 though, so in the meantime this is the easiest solution to keep it running.

Any update on this issue?

I've made a PR that tries to work around this problem. Would anyone be able to test it?
It's extremely ugly, so confirmation that it at least sort-of works from someone else would be reassuring :(
I've run the ruby-vips test suite with libvips 8.8, 8.9, 8.10 and 8.11, and tried @doits test program:
#!/usr/bin/ruby

require "vips"

::Vips::Image.new_from_array [[-1, -1, -1],
                              [-1, 32, -1],
                              [-1, -1, -1]], 24

@benubois With respect to https://github.com/janko/image_processing/issues/84, does this solve the issue? I am using jekyll_image_processing and blocked due to this.

@shreyas-sriram I re-ran my example with the fix-mutate-with-libvips8.9.1 branch and it did not trigger the issue I saw before.
Thanks @jcupitt!

Great! Thanks for testing!
I've merged to master, updated the autodocs, and pushed to rubygems as ruby-vips 2.1.1


Add Radarr webhook extras

The Radarr webhook currently only accounts for On Import
This issue is to enable the following additional notification triggers, to improve cache sync:

On Upgrade
On Movie Delete

@trueChazza Should these webhooks be added automatically by Midarr? The docs make it seem that way:

On server startup Midarr attempts to auto configure your integrations by:
Adding webhook / connect endpoints. This is for keeping your cache in sync

However I don't see any added in the Connect sections of Sonarr or Radarr

Interesting, thanks for reporting. I'll try a fresh install on my end and report back. There should at least be some logs saying it's attempted to do it.

https://github.com/midarrlabs/midarr-server/blob/feature/v3.0.0/lib/media_server/application.ex#L24
This is the "job" that auto starts with the application, and attempts to auto configure the webhook / connect endpoint in Radarr. It must be failing to start for some reason? I'll investigate.

@trueChazza Should these webhooks be added automatically by Midarr? The docs make it seem that way:

On server startup Midarr attempts to auto configure your integrations by:
Adding webhook / connect endpoints. This is for keeping your cache in sync

However I don't see any added in the Connect sections of Sonarr or Radarr, also I don't see anything in the Midarr logs related to this action...

Oh and yes it should be added automatically by Midarr.

 Yeah I can't see any mention of the webhook.

What Midarr version are you using?
What Radarr version are you using?

Midarr: 3.0.0 beta 9
Radarr: <IP_ADDRESS>56
Sonarr: <IP_ADDRESS>9

I wonder if I provide an example default so these jobs don't fail. Then we can override them?
Alternatively, I'd need to detail it out better in the README - in the upcoming Gotchas section maybe.
What do you think?

I wonder if I provide an example default so these jobs don't fail. Then we can override them?

Radarr / Sonarr both send a test POST when adding Connect endpoints, the URL must be valid. So this solution won't work.

Maybe just surfacing the error to the logs will be enough for now.

Yep that was it, I didn't have APP_URL set.


[SPARK-28798][FOLLOW-UP] Add alter view link to drop view

What changes were proposed in this pull request?
Add alter view link to drop view
Why are the changes needed?
create view has links to drop view and alter view
alter view has  links to create view and drop view
drop view currently doesn't have a link to alter view. I think it's better to link to alter view as well.
Does this PR introduce any user-facing change?
Yes
How was this patch tested?
Tested using jykyll build --serve

Test build #113674 has started for PR 26495 at commit f81b67a.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18557/
Test PASSed.

Test build #113674 has finished for PR 26495 at commit f81b67a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113674/
Test PASSed.

Merged to master

Thanks Sean!


Create Trigger Feed failing with Check syntax errors message

Hi I'm trying the sample code:
create trigger feed from alarm package
// for example...
const params = {cron: '*/8 * * * * *', trigger_payload: {name: 'James'}}
const name = '/whisk.system/alarms/alarm'
const trigger = 'alarmTrigger'
ow.feeds.create({name, trigger, params}).then(package => {
console.log('alarm trigger feed created')
}).catch(err => {
console.error('failed to create alarm trigger', err)
})
This is giving me a failure:
failed to create alarm trigger { BaseOperationError: Action invocation failed, API returned error code. Check syntax errors? Missing error message.
at Client.handle_errors (/home/lzhao/openwhisk/code/openwhisk-package-ecs/node_modules/openwhisk/lib/client.js:102:13)
at rp.catch.err (/home/lzhao/openwhisk/code/openwhisk-package-ecs/node_modules/openwhisk/lib/client.js:43:38)
at tryCatcher (/home/lzhao/openwhisk/code/openwhisk-package-ecs/node_modules/bluebird/js/main/util.js:26:23)
at Promise._settlePromiseFromHandler (/home/lzhao/openwhisk/code/openwhisk-package-ecs/node_modules/bluebird/js/main/promise.js:510:31)
at Promise._settlePromiseAt (/home/lzhao/openwhisk/code/openwhisk-package-ecs/node_modules/bluebird/js/main/promise.js:584:18)
at Promise._settlePromises (/home/lzhao/openwhisk/code/openwhisk-package-ecs/node_modules/bluebird/js/main/promise.js:700:14)
at Async._drainQueue (/home/lzhao/openwhisk/code/openwhisk-package-ecs/node_modules/bluebird/js/main/async.js:123:16)
at Async._drainQueues (/home/lzhao/openwhisk/code/openwhisk-package-ecs/node_modules/bluebird/js/main/async.js:133:10)
at Immediate.Async.drainQueues (/home/lzhao/openwhisk/code/openwhisk-package-ecs/node_modules/bluebird/js/main/async.js:15:14)
at runCallback (timers.js:666:20)
at tryOnImmediate (timers.js:639:5)
at processImmediate [as _immediateCallback] (timers.js:611:5)
name: 'BaseOperationError',
message: 'Action invocation failed, API returned error code. Check syntax errors? Missing error message.',
error:
{ duration: 109,
name: 'alarm',
subject: 'whisk.system',
activationId: '20ae80ca53c24fa3886fb9a23e2ba16c',
publish: false,
annotations: [ [Object], [Object] ],
version: '0.0.5',
response: { result: [Object], success: false, status: 'application error' },
end:<PHONE_NUMBER>980,
logs: [],
start:<PHONE_NUMBER>871,
namespace: 'whisk.system' } }
Can you let me know why this is not working? I am able to use the sample snippets to create an action without any problems but I'm not able to do it for an alarm

Is this an issue with the client or missing parameters to the feed? (@jasonpet FYI)

I'm just trying to test the sample code provided on the main page.  See full js file below (the commented out code works, I'm able to create the action without any problems, it's only the feed that doesn't work)  On the CLI i can also create the trigger no problem with the following command:
$ wsk trigger create alarmTrigger --feed /whisk.system/alarms/alarm -p cron "*/8 * * * * *"  -i
ok: invoked /whisk.system/alarms/alarm with id b56f1dca40154827ba75610eff1ba335
{
"activationId": "b56f1dca40154827ba75610eff1ba335",
"annotations": [
{
"key": "limits",
"value": {
"logs": 10,
"memory": 256,
"timeout": 60000
}
},
{
"key": "path",
"value": "whisk.system/alarms/alarm"
}
],
"duration": 62,
"end":<PHONE_NUMBER>615,
"logs": [],
"name": "alarm",
"namespace": "whisk.system",
"publish": false,
"response": {
"result": {},
"status": "success",
"success": true
},
"start":<PHONE_NUMBER>553,
"subject": "whisk.system",
"version": "0.0.5"
}
ok: created trigger alarmTrigger
Here's my full javascript file:
var fs = require('fs');
var openwhisk = require('openwhisk');
function action() {
var options = {apihost: '<IP_ADDRESS>', api_key: '789c46b1-71f6-4ed5-8c54-816aa4f8c502:abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP',ignore_certs: true};
var ow = openwhisk(options);
// for example...
const params = {cron: '*/8 * * * * *'}
const name = '/whisk.system/alarms/alarm'
const trigger = 'alarmTrigger'
ow.feeds.create({name, trigger, params}).then(package => {
console.log('alarm trigger feed created')
}).catch(err => {
console.error('failed to create alarm trigger', err)
});
//const actionName = 'pollAction'
//const action = fs.readFileSync('package.zip')
//ow.actions.create({actionName, action}).then(result => {
//  console.log('action created!')
//}).catch(err => {
//  console.error('failed to create action', err)
//})
}
action();
I also looked at the statusCode being returned and it's a 404 but i'm not sure why.
Here's logs from the docker container (for a successful use case) :
[2017-04-13T21:59:32.569Z] [INFO] [??] [alarmsTrigger] [POST /triggers] Got trigger {"name":"alarmTrigger","namespace":"","cron":"*/8 * * * * *","payload":{"payload":""}}
[2017-04-13T21:59:32.574Z] [INFO] [??] [alarmsTrigger] [POST /triggers] Checking if user has access rights to create a trigger
**[2017-04-13T21:59:32.590Z] [INFO] [??] [alarmsTrigger] [789c46b1-71f6-4ed5-8c54-816aa4f8c502] abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP https://<IP_ADDRESS>:443/api/v1/namespaces//triggers/alarmTrigger 200 user and password**
[2017-04-13T21:59:32.592Z] [INFO] [??] [alarmsTrigger] [createTrigger] 789c46b1-71f6-4ed5-8c54-816aa4f8c502:abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP//alarmTrigger starting cron job
[2017-04-13T21:59:32.609Z] [INFO] [??] [alarmsTrigger] [POST /triggers] 789c46b1-71f6-4ed5-8c54-816aa4f8c502:abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP//alarmTrigger created successfully
For the one I'm invoking via code 👍
2017-04-13T21:54:14.516Z] [INFO] [??] [alarmsTrigger] [POST /triggers] Got trigger {"name":"alarmTrigger","namespace":"","cron":"*/8 * * * * *","payload":{"payload":""}}
[2017-04-13T21:54:14.517Z] [INFO] [??] [alarmsTrigger] [POST /triggers] Checking if user has access rights to create a trigger
**[2017-04-13T21:54:14.539Z] [INFO] [??] [alarmsTrigger] [789c46b1-71f6-4ed5-8c54-816aa4f8c502] abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP https://<IP_ADDRESS>:443/api/v1/namespaces//triggers/alarmTrigger 404 user and password**
[2017-04-13T21:54:14.540Z] [ERROR] [??] [alarmsTrigger] [POST /triggers] Trigger authentication request failed.

let me know if you need additional logs or information

Have you already created the trigger (alarmTrigger) that this feed relies on?
This error message is consistent with the feed referencing a trigger that does not exist.
The error message returned is not helpful to understanding when this error occurs. We have an open issue (#45) to improve error messages returned from the API. I'll add this as an example to investigate.
I'll update the README.md to clarify this dependency.

@jthomas Yes, I figured out the same thing later that day when I printed out the status code returned and tested some code to add a trigger first.  That's exactly what was wrong with my code.  I needed to create the trigger associated with my feed in my code first.  Thanks for your help.


Ignore Annotation for Methods

Would you be open to a PR that adds an @Ignore annotation that AVM would use to discard annotated fields in its generated adapter? AFAIK this doesn't exist.
Related Moshi issue here #94.

@rharter and I have talked in the past about making a public annotations artifact with some common serialization annotations (since av-gson has a form of this already). I think we should just do that rather than fragment further, then av-moshi can just support the same annotations.

You could use @Transient from the lazy adapters. Also there's SerializeOnly as well as DeserializeOnly.

I'm not sure we'd want to pull that library into AVM, the surface area looks pretty large. If the annotations were separate that might make sense.

You don't need to add this in here. It'll work with autovalue out of the box. You can see that in the integration tests.

Yep I think that's actually a better recommended solution. I forgot that existed. Pretty much a perfect example of moshi's annotation support, and doesn't require this library to know anything about it.

Ah great. I'll close this out then. Thanks @vanniktech and @hzsweers.


remove unnecessary Gradle wrapper configuration

wrapper is an implicit Gradle task, you don't need to create one manually, just call e.g.
./gradlew wrapper --gradle-version=2.12

not sure about this commit. You are right in that there is a builtin wrapper task and that our wrapper task should not be needed. But the soruce distribution comes without the gradlew files. So how does a user know what gradle version to use to ensure there are no problems with the built related to the built tool version? Right now we can tell people to use any gradle to execute the wrapper task and then work with the wrappers from there on

Thanks for the feedback. Actually I forgot about the source distribution. Why does it miss the gradle wrapper at all?
Now that you mentioned it, one of my previous pull requests removed the declaration of wrapper generation as default task in Gradle, thereby breaking the documentation describing the bootstrap of the source distribution. We should either

include the Gradle wrapper in the source distribution, or
change the README.adoc to call gradle wrapper instead of gradle.

-1 on this change. Apache source distro's can't have the wrapper jar (a binary). They make some minor exceptions but to date the gradle wrapper isn't one of the allowable exceptions. The downloadWrapper task was meant to be a default task but that must have been overlooked in earlier commits. I just fixed that, so now the doco should be correct. This still leaves the implicit wrapper task with an explicit version free should someone wish to not abide by our recommendations.

Thanks for the explanation.
I still think it is a good idea to not use a default task, so that Gradle prints a helpful splash screen when invoked without arguments. I propose to document the use of the implicit Gradle wrapper task. Its version can still be overridden on the command line as described in the Gradle manual. I may also link to the Gradle documentation describing this use case in the README.adoc if desired.

rebased from master

This looks better. Nice work. I'll just double check that everything works as expected before merging.

Looks like you've deleted the repo. Okay if I apply manually?

Please feel free 👍

merged, thanks!


add support for defining multiple targets in CreateLibrary

allow rocBLAS to define multiple targets for CreateLibrary

Seems like you are making a list with lists inside. Did you try
COMMAND "${Tensile_CREATE_MANIFEST_COMMAND"
Sorry but could be your original workaround is better if that fails, quirky cmake, this might be much better handled directly as python down the road.

Unfortunately that fails due to us needing to expand the command prior to sending it into the command.


irmin-fsck: traverse head commit for stats

Added a new subcommand in irmin-fsck which traverses the graph under one commit and reports: the longest paths, the widest nodes, and the "biggest" Merklee proof = the chain from root to leaf that stores the most hashes, where for each node we store its siblings. For the first two, we need to visit all nodes, but for the MP we need to visit all inodes as well. I think this is what we want to compare inode configurations, let me know if I got it wrong.
It also prints all paths in a separate file. It's still draft because I'm not sure how we want to print/report these.

This is rebased and can be reviewed. Here is the output from a very recent snapshot:
> time PACK_LAYERED=false dune exec -- ./test/irmin-pack/cli/irmin_fsck.exe stat-store ~/Documents/tezos/data/context --commit CoWWJ4hjcS3jG3a11FX3gnKPiY5HpfrCsGCwFC3uRwsKJeR4imYm --all_paths=all_paths_large
>> Max width is 256 for 5451 nodes, an example: data/commitments
>> Max MP is 231 for 6 blobs, an example: data/contracts/index/-/1b/-/45/9f/60/b8/f0/0000a239f27133bef5fb66577f05f2f62c59423ab1df/delegated/-/0c/-/83/54/38/51/c4/00004933bacf709c9b715bb61ddee1dfcc16af387e05
>> Max length is 18 for 4149889 paths, an example: data/big_maps/index/eb/86/49/21/49/97/52/contents/4a/d4/d4/cf/52/0bc6758e96edefbdef8ae7587cb90a9b55df5a02a32b1db6ae8544/data

________________________________________________________
Executed in  691.24 secs

> head all_paths_large
data/contracts/index/cf/1c/cd/f1/f0/d6/00002ca13af0253e78f07edf95726e332c0d44add593/manager
data/contracts/index/23/15/80/ae/89/ae/01b8acea81ebb8c1717bbeb1e2ead27b1043676ba900/data/storage
data/contracts/index/e8/c2/9d/06/9b/27/0000e8498e1510e3866990c6b9848e6d1a94785eeb44/balance
data/contracts/index/6f/78/f7/6b/76/11/00004ca4e200935897001a3cc3b44c727b238135d764/manager
...


#134 Improve watch defaultValues and getValues with non flatten data structure

[x] defaultValues inject default value into watch
[x] improve watch with array and objects
[x]  getValues with arugment to nest, default nest to false (similar functionality will apply to watch)
[x] defaultValues support array and objects
[ ] update website with getValues update
[ ] update website with defaultValues and defaultValue association with watch

watch API update:

watch return default value change:
watch(['test', 'test1']) // deafult value return [undefined, undefined] instead of undefined
watch() // default value return {} instead of undefined

watch integrate with defaultValues
watch will inherit from data in defaultValues, but watch function arguments remain unchanged. second argument defaultValue will overwrite data in defaultValues which set up in useForm()

watch now support nested object look up
eg

  const data = watch('test');
  console.log(data);

    <form>
      <input name="test[0]" ref={register} />
      <input name="test[1]" ref={register} />
    </form>

this above example will watch both test[0] and test[1]
defaultValues update:
defaultValues now support nested object, eg:
const { register } = useForm({
  defaultValues: { test: {data: [1, 2]}},
})

getValues update:
getValues now support nest argument to return data in a nested format, eg:
getValues(); // {'test[0]': 1, 'test[1]': 2}
getValues({ nest: true }); // {test: [1,2]}

@barrymay @stramel wondering if I can get you guys to help on typescript part, I still have couple @ts-ignore  in the repo.

I have released a beta version if you guys want to help on testing<EMAIL_ADDRESS>much appreciated

Will test in a few and will try to get rid of the ts-ignores

@bluebill1049 Overall seems to work great!

getValues currently requires a payload. We should make that optional.
errors types that I updated, make the DX worse so we may want to update that eventually to make it better. Also, they still need to be updated to match the actual flattened structure.
watch api: What's the benefit of the defaultValue override? If we drop that, we could clean up to accept any number of properties. watch('attributes', 'name', 'age', 'type').

@stramel that's for the testing. ❤

getValues currently requires a payload. We should make that optional.
agree! my mistake

watch api: What's the benefit of the defaultValue override? If we drop that, we could clean up to accept any number of properties. watch('attributes', 'name', 'age', 'type').
I didn't want to make a breaking change, and if people prefer to use defaultValue on the input itself. then you will need a defaultValue set at watch

WRT @ts-ignores I was thinking about tackling them after this PR goes through.

@bluebill1049

getValues currently requires a payload. We should make that optional.

agree! my mistake

All good! Just something I caught

errors types that I updated, make the DX worse so we may want to update that eventually to make it better. Also, they still need to be updated to match the actual flattened structure.

:(

Yeah, I can try to fix it with some of the typing updates.

watch api: What's the benefit of the defaultValue override? If we drop that, we could clean up to accept any number of properties. watch('attributes', 'name', 'age', 'type').

I didn't want to make a breaking change, and if people prefer to use defaultValue on the input itself. then you will need a defaultValue set at watch

That makes sense! I wonder if it is worth supporting the defaultValue on watch when the input itself could handle defaultValue? I suppose supporting it through watch allows for more generic handling and not requiring a component that has defaultValue.

That makes sense! I wonder if it is worth supporting the defaultValue on watch when the input itself could handle defaultValue? I suppose supporting it through watch allows for more generic handling and not requiring a component that has defaultValue.

it's only made for the first render, otherwise watch() will return undefined, because input hasn't been register
const toggle = watch('test') //first render will be undefined, defaultValue can prevent this to happen

// const toggle = watch('test', 'haha') can solve this problem.

<input name="test" defaultValue="haha" />

{toggle && 'showme'}


Raspberry PI Camera Demo - Added Note to Step 9

I added a note to step 9 as missing this caused an afternoon of searching for resolutions

@WaywardHayward this is great. Just need to sign the commits and its good to go. To sign:
git rebase HEAD~2 --signoff
git push --force-with-lease origin main


Google-POI: Category mapping

The pipeline should also convert the Google category tags into the same high-level categories as it is done in the OSM-POI pipeline. This makes merging the data sources possible also on the category level and enables queries based on categories for Google POIs.

assign me. :)

Here is some more info about the requirements. You can basically orient yourself along the osm-poi pipeline. Even though it's written in JS, you can easily transfer the functionality to Python.
You can find the category mapping for the OSM POI categories under kuwala/kuwala-pipelines/osm-poi/resources/categories.json. For the Google POI category mapping, you should create a file with the same structure (the same high-level categories) under kuwala/kuwala-pipelines/google-poi/resources/categories.json.
In kuwala/kuwala-pipelines/osm-poi/src/data/processor/pbfParser/tagParser/tagParser.js, we have already a mapping function written in JS.
We already retrieve the unmapped category tags from Google in kuwala/kuwala-pipelines/google-poi/src/routes/poi_information.py. The mapped categories should be stored in a separate property to do more granular mapping in the future possibly.
To generate more examples, we suggest using Postman. We don't expect you to map hundreds of category tags. If a tag doesn't exist in a high-level category, it should be added under misc in the categories.json so it is possible to attribute it to a fitting high-level category later on.


Implement full Takeout folder migration

This MR adds the flag fullMigrate (as opposed to folderMigrate which implements the old functionality of folder -> folder migration), which automatically migrates an entire takeout file into AlbumsProcessed and PhotosProcessed which have all metadata sorted and can easily migrated to any platform.
This addresses #6
Things to keep in mind:

I did not implement a --force flag to fullMigrate. I personally don't think you should be forcing a large-scale migration like this. If this is an issue it can be added back.
I tried to make this run in the least destructive way possible in case things go wrong. But because of this, there will be 3 copies of your photos in the end. The origional unstructured photos and albums in the Google Photos directory, the sorted photos and albums in the Google Photos/Albums & Google Photos/Photos, and then the final processed & sorted copy in the Google Photos/AlbumsProcessed & Google Photos/PhotosProcessed. If this is an issue, I can add a --destructive flag which moves directories instead of copying them, as well as deleting a directory after processed all of the photos inside it.

Ah I messed up removing the build/ dir. I'll fix that and then update this comment.

Wow that made it way worse lol

@lukehmcc The README in your branch has the wrong instructions for your new command.  It seems to need fullMigrate instead of migrateFolder, but I only figured that out by reading here.
The help text of the tool itself also makes no mention of the subcommands, it only prints the help text for the old method, which now also needs the migrateFolder subcommand.

It looks like your cleanup error handling still has some bugs too.  After the initial processing is done, I'm getting output like this on my photo set:
Done! Processed 11256 files.
Files migrated: 11146
Files failed: 110
Rewriting all tags from /mnt/Google Photos/PhotosError/original_8785a13f-e6d2-45ce-a096-0c237dad625e_2(1).jpg, to  /mnt/Google Photos/Photos/cleaned-original_8785a13f-e6d2-45ce-a096-0c237dad625e_2(1).jpg
Cannot fix metadata for /mnt/Google Photos/PhotosError/iMarkup_20190704_122554.png.json as .json is an unsupported file type.
Rewriting all tags from /mnt/Google Photos/PhotosError/iMarkup_20190704_122554.png, to  /mnt/Google Photos/Photos/cleaned-iMarkup_20190704_122554.png
Rewriting all tags from /mnt/Google Photos/PhotosError/duplicates-2, to  /mnt/Google Photos/Photos/cleaned-duplicates-2
/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41
                    error = new Error(errMsg);
                            ^

Error: '/mnt/Google Photos/Photos/cleaned-duplicates-2' already exists - /mnt/Google Photos/PhotosError/duplicates-2/20230224_131215.jpg
    at RewriteAllTagsTask.parser (/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29)
    at RewriteAllTagsTask._Task_resolve (/app/node_modules/batch-cluster/dist/Task.js:146:40)

It kind of looks like the rewriteAllTags that you automatically run after the restructuring and renaming isn't properly filtering out the *.json files.  Keep in mind that takeout can end up with *.json files named with or without the file extension of the file they're associated with, so you can't just look for .png, you have to look for .png$ (it has to be at the end of the file name).
It also looks like there's something with the duplicates to cleaned-duplicates that's either running more times than it should on the same file, isn't picking properly unique file names, or is erroneously blocking overwrites of the cleaned-duplicates file.

Also fun fact, you seem to have accidentally messed up what folder you're putting Albums and Photos in when you restructure the folder.  It's putting it into /path/to/takeout/Google Photos/{Albums,Photos} instead of the /path/to/takeout/{Albums,Photos} you error check at the start.
[...snip...]
Copying /mnt/Google Photos/Photos from 2023 to /mnt/Google Photos/Photos/Photos from 2023
[...snip...]
Copying /mnt/Google Photos/Wedding to /mnt/Google Photos/Albums/Wedding

@lukehmcc The README in your branch has the wrong instructions for your new command. It seems to need fullMigrate instead of migrateFolder, but I only figured that out by reading here.
~The help text of the tool itself also makes no mention of the subcommands, it only prints the help text for the old method, which now also needs the migrateFolder subcommand.~
EDIT: I was accidentally running help on a subcommand and didn't realize it.

Good catch on the documentation issue.

It looks like your cleanup error handling still has some bugs too. After the initial processing is done, I'm getting output like this on my photo set:
Done! Processed 11256 files.
Files migrated: 11146
Files failed: 110
Rewriting all tags from /mnt/Google Photos/PhotosError/original_8785a13f-e6d2-45ce-a096-0c237dad625e_2(1).jpg, to  /mnt/Google Photos/Photos/cleaned-original_8785a13f-e6d2-45ce-a096-0c237dad625e_2(1).jpg
Cannot fix metadata for /mnt/Google Photos/PhotosError/iMarkup_20190704_122554.png.json as .json is an unsupported file type.
Rewriting all tags from /mnt/Google Photos/PhotosError/iMarkup_20190704_122554.png, to  /mnt/Google Photos/Photos/cleaned-iMarkup_20190704_122554.png
Rewriting all tags from /mnt/Google Photos/PhotosError/duplicates-2, to  /mnt/Google Photos/Photos/cleaned-duplicates-2
/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41
                    error = new Error(errMsg);
                            ^

Error: '/mnt/Google Photos/Photos/cleaned-duplicates-2' already exists - /mnt/Google Photos/PhotosError/duplicates-2/20230224_131215.jpg
    at RewriteAllTagsTask.parser (/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:41:29)
    at RewriteAllTagsTask._Task_resolve (/app/node_modules/batch-cluster/dist/Task.js:146:40)

It kind of looks like the rewriteAllTags that you automatically run after the restructuring and renaming isn't properly filtering out the *.json files. Keep in mind that takeout can end up with *.json files named with or without the file extension of the file they're associated with, so you can't just look for .png, you have to look for .png$ (it has to be at the end of the file name).
It also looks like there's something with the duplicates to cleaned-duplicates that's either running more times than it should on the same file, isn't picking properly unique file names, or is erroneously blocking overwrites of the cleaned-duplicates file.
EDIT:
FYI, I cloned your exif-wrapper repo, changed the submodule to use your fork on this branch, built the Dockerfile, and then manually entered the resulting container with my takeout folder host-mounted at /mnt. I'm running all these commands from within the container, so the tool code being run is at /app.

It already checks for json endings before running rewriteAllTags.
https://github.com/lukehmcc/google-photos-migrate/blob/master/src/cli.ts#L84
I'm not sure why you're running into this.
Also please do not use exif-wrapper, I'm migrating all of it's functionality to this repo with the the fullMigrate command. exif-wrapper is currently broken.

Also fun fact, you seem to have accidentally messed up what folder you're putting Albums and Photos in when you restructure the folder. It's putting it into /path/to/takeout/Google Photos/{Albums,Photos} instead of the /path/to/takeout/{Albums,Photos} you error check at the start.
[...snip...]
Copying /mnt/Google Photos/Photos from 2023 to /mnt/Google Photos/Photos/Photos from 2023
[...snip...]
Copying /mnt/Google Photos/Wedding to /mnt/Google Photos/Albums/Wedding

Though based on your first comment in this PR, maybe you just messed up which folders you're error checking at the start of the command instead.

I don't see how this is a mistake? It moved the album to the album directory and the photos to the photo directory...

Also please do not use exif-wrapper, I'm migrating all of it's functionality to this repo with the the fullMigrate command. exif-wrapper is currently broken.

Yeah, I realized that. It would probably be more accurate to say I used your Dockerfile to create an npm environment with this tool in it. I didn't try your python script.

Also fun fact, you seem to have accidentally messed up what folder you're putting Albums and Photos in when you restructure the folder. It's putting it into /path/to/takeout/Google Photos/{Albums,Photos} instead of the /path/to/takeout/{Albums,Photos} you error check at the start.
[...snip...]
Copying /mnt/Google Photos/Photos from 2023 to /mnt/Google Photos/Photos/Photos from 2023
[...snip...]
Copying /mnt/Google Photos/Wedding to /mnt/Google Photos/Albums/Wedding

Though based on your first comment in this PR, maybe you just messed up which folders you're error checking at the start of the command instead.

I don't see how this is a mistake? It moved the album to the album directory and the photos to the photo directory...

I realized the mistake is actually which directory you were error checking for earlier. I misunderstood which was correct and which wasn't.

@mtalexan
Okay that should be better. I just did a test and it successfully processed & organized ~10k assets. Lmk how it goes for you.


Allow the selection of multiple files in the file extension

Before anyone says, anything, I am aware you can the zip extension to add multiple files at once. My main issue with this is that if you wanted the end user to be able to import multiple files at once, they would have to import it one at a time or as a zip file, which is very counter intuitive and clunky.
Also, I am aware of Issue #347, but the creator of the issue closed it as he could use a zip file.
We would of course have it as a separate block, and all in all I feel like it's probably relatively highly requested feature.

i suppose the question is how to design the blocks to make this work

We could essentially make it like the zip extension, where it has it's own file system, but that would require a whole rewrite and probably isn't the best way to do it.
I was thinking about it, and maybe 3 blocks

Open multiple files
"Name of opened files" (as an array)
Contents of file (name)
We would probably have one "set" of cached files at a time

i suppose the question is how to design the blocks to make this work easily

you could replace the extension with my Files Expanded. Basically runs TW files but with enhancements such as that


In-line webview on iOS camera permission

I've granted the permissions for camera/microphone access (asked permission, add relevant keys in info.plist), but the webpage I load still doesn't have access to camera.
Here is how I build the widget:
InAppWebView(
        initialUrl: model.param.url,
        initialOptions: InAppWebViewGroupOptions(
          crossPlatform: InAppWebViewOptions(
            useShouldOverrideUrlLoading: true,
            mediaPlaybackRequiresUserGesture: false,
          ),
          android: AndroidInAppWebViewOptions(
            useShouldInterceptRequest: true,
          ),
          ios: IOSInAppWebViewOptions(
            allowsInlineMediaPlayback: true,
          ),
        ),
        onLoadStart: (controller, url) {
          model.onUrlLoaded(url);
        },
        shouldOverrideUrlLoading: (controller, shouldOverrideUrlLoadingRequest) async {
          return ShouldOverrideUrlLoadingAction.ALLOW;
        },
        androidOnPermissionRequest: (controller, origin, resources) async {
          return PermissionRequestResponse(resources: resources, action: PermissionRequestResponseAction.GRANT);
        },
      );

I'm running on iOS 13.3, Flutter 1.22.6. Using the plugin version flutter_inappwebview: ^4.0.0

Check the WebRTC and  WebRTC on iOS sections to be sure you have added all the necessary permissions.
Other than that, there is nothing more to add.

iOS v15.5
flutter v3.0
One of the application widgets uses InAppWebView with a page that is using camera/micro.
return Scaffold(
    appBar: AppBar(
    title: const Text('XPTO'),),
    body: SafeArea(
      child: Column(children: <Widget>[
        Expanded(
          child: Stack(
            children: [
              InAppWebView(
                key: webViewKey,
                initialUrlRequest: URLRequest(url: Uri.parse(app.url.toString())),
                initialOptions: options,
                androidOnPermissionRequest: (InAppWebViewController controller, String origin, List<String> resources) async {
                  return PermissionRequestResponse(resources: resources, action: PermissionRequestResponseAction.GRANT);
                },
                onWebViewCreated: (controller) async  {
                  webViewController = controller;
                },
          )])
        )
      ]
  )
));

the options object for the InAppWebView is:
InAppWebViewGroupOptions options = InAppWebViewGroupOptions(
  crossPlatform: InAppWebViewOptions(
    mediaPlaybackRequiresUserGesture: false,
    javaScriptEnabled: true
  ),
  ios: IOSInAppWebViewOptions(
    allowsInlineMediaPlayback: true,
    allowsAirPlayForMediaPlayback: true
  ));
On main.dart I've put the permission request

  WidgetsFlutterBinding.ensureInitialized();

  await Permission.camera.request();
  await Permission.microphone.request();
On Inflo.plist

<key>NSCameraUsageDescription</key>
<string>This thing use camera</string>
<key>NSMicrophoneUsageDescription</key>
<string>This thing use the microfone</string>
<key>UILaunchStoryboardName</key>

On pod:
        ## dart: PermissionGroup.camera
        'PERMISSION_CAMERA=1',

        ## dart: PermissionGroup.microphone
        'PERMISSION_MICROPHONE=1',

Everything is working fine.
The problem is that every time i open the InAppWebView, the message "Allow ... to use your camera and microphone?" appears.
as the image

Do you happen to know how to prevent that permission dialog message to show up?

iOS v15.5 flutter v3.0
One of the application widgets uses InAppWebView with a page that is using camera/micro.
return Scaffold(
    appBar: AppBar(
    title: const Text('XPTO'),),
    body: SafeArea(
      child: Column(children: <Widget>[
        Expanded(
          child: Stack(
            children: [
              InAppWebView(
                key: webViewKey,
                initialUrlRequest: URLRequest(url: Uri.parse(app.url.toString())),
                initialOptions: options,
                androidOnPermissionRequest: (InAppWebViewController controller, String origin, List<String> resources) async {
                  return PermissionRequestResponse(resources: resources, action: PermissionRequestResponseAction.GRANT);
                },
                onWebViewCreated: (controller) async  {
                  webViewController = controller;
                },
          )])
        )
      ]
  )
));

the options object for the InAppWebView is:
InAppWebViewGroupOptions options = InAppWebViewGroupOptions(
  crossPlatform: InAppWebViewOptions(
    mediaPlaybackRequiresUserGesture: false,
    javaScriptEnabled: true
  ),
  ios: IOSInAppWebViewOptions(
    allowsInlineMediaPlayback: true,
    allowsAirPlayForMediaPlayback: true
  ));
On main.dart I've put the permission request

  WidgetsFlutterBinding.ensureInitialized();

  await Permission.camera.request();
  await Permission.microphone.request();
On Inflo.plist

<key>NSCameraUsageDescription</key>
<string>This thing use camera</string>
<key>NSMicrophoneUsageDescription</key>
<string>This thing use the microfone</string>
<key>UILaunchStoryboardName</key>

On pod:
        ## dart: PermissionGroup.camera
        'PERMISSION_CAMERA=1',

        ## dart: PermissionGroup.microphone
        'PERMISSION_MICROPHONE=1',

Everything is working fine.
The problem is that every time i open the InAppWebView, the message "Allow ... to use your camera and microphone?" appears.
as the image

Do you happen to know how to prevent that permission dialog message to show up?

I followed the docs given, everything works fine in android.
GPS and camera permission works as expected.
but in iOS, they always reprompting the permission for gps and camera everytime the apps want to access that interfaces.
if someone found a solution, really appreciate if you can share it.
Thanks


New template for reimbursement contract

As NHS Admin i want this (https://drive.google.com/file/d/1YFvHYsOuxc-gQjIu7dDy1ni0vwWL2yU8/view?usp=sharing)  template to be used for all reimbursement contracts so contracts will comply with legal requirements

the link to contract template has been added


Update project URL to new https://pypi.org/project/setuptools/

On the GitHub brief project description/home page:
https://github.com/pypa/setuptools
The URL uses the old PyPI:
https://pypi.python.org/pypi/setuptools
And should be:
https://pypi.org/project/setuptools/

@stevepiercy Thanks for noticing! I've fixed this.


Plunker CPU usage in Google Chrome hits 100%

Hey guys, I'm getting this issue on my mid 2014 Macbook Pro using Google Chrome.
I've disabled the option "Use hardware acceleration when available", but still I get high CPU usage and the fan starts going crazy!

I'm facing the same issue too, if you have how to calm it down please help. Thanks

Same issue.


Need demo of data pipeline connected to ECB RESTful interface (exchange rates and more)

The ECB makes datasets available via a RESTful interface: https://sdw-wsrest.ecb.europa.eu/help/
From that help page there is an example of how to format requests to download datasets such as USD/EUR exchange rates (the .A at the end is Annual; Quarterly, Monthly, Daily rates are also available): https://sdw-wsrest.ecb.europa.eu/service/data/EXR/M.USD.EUR.SP00.A
Ideal would be an initial implementation that could be generalized into a federating service of all publicly available ECB statistics, using features of their RESTful interface to guide data discovery and selection.  Exchange rate data is needed immediately for ITR and PCAF workstreams.  Later enhancements can cache data to Trino and update as needed.
@HeatherAck @toki8 @caldeirav @erikerlandson @LeylaJavadova @mersin35

This is link syntax to get CSV data instead of XML: https://sdw-wsrest.ecb.europa.eu/service/data/EXR/M.USD.EUR.SP00.A?format=csvdata

@bryonbaker as discussed, we can use this issue to build our first REST-based integration. We can start with daily exchange rates.

I have a few initial questions...

Are we looking for a live stream, or something that will create a historical feed?
Are we starting with just FX?
Will this include stocks of interest?
Has there been discussion on which components we will incorporate in the solution? E.g. service pulls feeds from markets and dumps them on a kafka topic or persist them in a database, .....

@bryonbaker architecturally speaking, this is the pattern we want to demonstrate: https://github.com/os-climate/os_c_data_commons/blob/main/docs/add-data-pipeline-ingestion-patterns-event.md

Let's just start with FX, polish that, then extend.
I get GitHub notifications on my phone.  OS-Climate email is also good for things that don't (yet) tie well to issues.

Suggested core requirements - please provide feedback:

Retrieve FX prices from one or more markets and publish these on an event stream (kafka topic)
Frequency / scheduling is configurable per market
Proposed JSON-format schema for the event-stream data is below. This is based on a schema from OANDA that I was playing with.

{     "base_currency": "USD",     "meta": {         "effective_params": {             "data_set": "OANDA",             "date": "2022-04-20",             "decimal_places": "5",             "fields": [                 "averages",                 "highs",                 "lows",                 "midpoint"             ],             "quote_currencies": [                 "AUD",                 "GBP"             ]         },         "request_time": "2022-04-20T01:58:29+0000",         "skipped_currencies": []     },     "quotes": {         "AUD": {             "ask": "1.35650",             "bid": "1.35622",             "date": "2022-04-19T23:59:59+0000",             "high_ask": "1.36154",             "high_bid": "1.36127",             "low_ask": "1.35152",             "low_bid": "1.35124",             "midpoint": "1.35636"         },         "GBP": {             "ask": "0.76894",             "bid": "0.76881",             "date": "2022-04-19T23:59:59+0000",             "high_ask": "0.77038",             "high_bid": "0.77027",             "low_ask": "0.76688",             "low_bid": "0.76675",             "midpoint": "0.76888"         }     } }

As a clarification, the use of event streams is suggested because of the need for REST API integration, not because of real time data streaming requirements. If we rely on API calls for the source data, we need a way to load source data in a persisting manner for each API call in order to be have a fully reproducible pipeline in the future (i.e. another API call may not yield the same data). By persisting the result of the API calls in streams, we are able to have an end-to-end fully reproducible data pipeline (we just need to republish the stream).

I thought the idea was to use kafka as a time series database to enable it an an input to more than reporting such as machine learning.
I am fine with either approach, but I need clarity on the requirements so I can start designing.

Yes we would use kafka as a time series database indeed (what i describe as persisting the result of the API calls in streams). So multiple pipelines consuming some historical rates can consume these via Trino, with full reproducibility.

@bryonbaker Could you provide a progress update on this issue at the Data Commons tech check-in meeting tomorrow?

@bryonbaker built a standardised interface service we can plug into multiple price sources and feed data into Strimzi:
https://github.com/bryonbaker/markets-pricing-service
I will be testing this while we wait for the deployment of Strimzi to be on osc-cluster2. @redmikhail for your information as we have a dependency on this setup

Looks like https://github.com/os-climate/os_c_data_commons/issues/160 was closed, so we can now close this, yes?

Which source tree should I build the pricing service? So you get idea of what is being built, this is the initial version I have built for the market simulator in my own repo:
https://github.com/bryonbaker/markets-pricing-service

@MichaelTiemann - sorry I missed your request above "@bryonbaker Could you provide a progress update on this issue at the Data Commons tech check-in meeting tomorrow?"
Happy to give an update at the next one. This is my first Git Issue for OS Climate. What kind of update is required?

@mersin35 has been working on the this repo: https://github.com/os-climate/PCAF-sovereign-footprint
It's received a lot of positive attention, and one of the last steps is to demonstrate access to a federate source for FX rates.  Ideally we can replace this approach (https://github.com/os-climate/PCAF-sovereign-footprint/blob/master/notebooks/OECD-EXCH-ingest.ipynb) with an API to the ECB data.  The goal of the update would be to provide a warm handoff of what's working RESTfully to him (and to me--I do FX stuff with the SEC DERA data).

The market pricing service I built is easily extensible to add an api endpoint. It was designed to be completely pluggable for market data providers and market data publishers (kafka, api, console, sql, ...)

We should create a brand new repo for ELT of real time market data including FX Pricing data as this is going to be used by multiple pipelines as reference data.
I suggest creating a new repo e.g. market-data-ingestion-service under OS-Climate GitHub organization where we have:

The base code for the service which is markets-pricing-service
Additional code required for the ECB data provider integration (note: this is not really a real time provider as these are reference rates, so need some logic in this case to capture only reference rates once)
In the future any provider that needs to be added should have a structure ready and documented for contributors to the repo

The above will give us extraction / loading for rates straight into Trino which is how we want to do federation in the case of API-based integration. If need for additional data transformation on source data, DBT models and pipelines can also be added as an extension to the repo. Transformation required by specific pipelines (like PCAF) should be done in their own pipeline repo.

We agreed that data for this use-case the trino schema will be tied to the provider so we can have multiple FX providers streaming based on their raw format into kafka, based on their own schema, and the ECB provider source would be polled on a regular cadence (daily) through some automation.

Need to check with Bryon  to see if he is ready to push the ECB @bryonbaker

I am still in Dev. I am retrieving the FX details and just getting it ready to Marshall onto Kafka. I need a few more days before it is ready in Dev.

I am reading the FX data from ECB now. The json format is   v e r y   complex. Please review and comment - are you going to be able to handle this? Here is an example of yesterday's FX for AUD against EUR: https://gist.github.com/bryonbaker/24b4df73d8cafe6e26b342094b681bd2

Internal
Hi Bryan,
thanks for your work. Mutlu Ersin, who is working on this piece, is currently on vacation. Back next week.
Best Regards,
Thomas
Thomas Kirchherr
Head of Sales and Account Management
<EMAIL_ADDRESS>P +49 89 3800 15117
M +49<PHONE_NUMBER>
Be green, keep it on the screen.
Von: Bryan Baker @.>
Gesendet: Dienstag, 30. August 2022 14:21
An: os-climate/data-platform-demo @.>
Cc: Kirchherr, Thomas (IDS GmbH) @.>; Mention @.>
Betreff: [EXT] Re: [os-climate/data-platform-demo] Need demo of data pipeline connected to ECB RESTful interface (exchange rates and more) (Issue #51)
I am reading the FX data from ECB now. The json format is v e r y complex. Please review and comment - are you going to be able to handle this? Here is an example of yesterday's FX for AUD against EUR: https://gist.github.com/bryonbaker/24b4df73d8cafe6e26b342094b681bd2
—
Reply to this email directly, view it on GitHubhttps://github.com/os-climate/data-platform-demo/issues/51#issuecomment-1231590142, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASV2E7QIBQZQAZDWX5IOIQTV3X4BTANCNFSM5XH4S6EQ.
You are receiving this because you were mentioned.Message ID<EMAIL_ADDRESS>

I discussed this with Vincent and here is the plan of approach...
Originally the idea was to take the entire message from the data source and put on the topic, where data pipelines can pick up all the data and do what they want with it. Prune, combine, join other data sources, ... But when the message is as complicated as this, that approach is not going to work well.
So I will extract the salient data that we want for OS Climate and put that in a new message structure, with a well-defined Trino schema.
Next Steps:

I will identify the attributes to extract and propose the schema in this Issue and get agreement from the team.
Implement and publish

@MichaelTiemann - I am almost there. A few questions:

How far back in time do you want the data?
What do you want the base currency to be for the initial load?
Which currencies do you want included?

Thanks

I think 1990 is far enough back.

@MichaelTiemann - thanks. Will do.
And the other two questions? Which currencies and which base currency (EUR?)
Thanks

I presume that we can change things as needed.  But in the mean time the pint demo for Iceberg (https://github.com/os-climate/data-platform-demo/blob/iceberg/notebooks/pint-demo.ipynb) defines these currencies:
ureg.define("USD = [currency] = $")
ureg.define("EUR = [currency_EUR] = €")
ureg.define("GBP = [currency_GBP] = £")
ureg.define("JPY = [currency_JPY] = ¥")
ureg.define("KRW = [currency_KRW] = ₩")
ureg.define("UAH = [currency_UAH] = ₴")
ureg.define("THB = [currency_THB] = ฿")
ureg.define("TRY = [currency_TRY] = ₺")
ureg.define("INR = [currency_INR] = ₨ = ₹")
ureg.define("MXN = [currency_MXN] = ₱")
ureg.define("ILS = [currency_ILS] = ₪")
ureg.define("AUD = [currency_AUD] = ₳")
ureg.define("RUB = [currency_RUB] = ₽")

We could start with those and update as needed.  As for base currency, let's start with EUR and change if needed.  I can fix the pint demo to adapt ;-)

Yes. We can adjust. I just want to load all the essentials from day 1
On Sun, 11 Sep 2022 at 6:22 pm, Michael Tiemann @.***>
wrote:

I presume that we can change things as needed. But in the mean time the
pint demo for Iceberg (
https://github.com/os-climate/data-platform-demo/blob/iceberg/notebooks/pint-demo.ipynb)
defines these currencies:
ureg.define("USD = [currency] = $")
ureg.define("EUR = [currency_EUR] = €")
ureg.define("GBP = [currency_GBP] = £")
ureg.define("JPY = [currency_JPY] = ¥")
ureg.define("KRW = [currency_KRW] = ₩")
ureg.define("UAH = [currency_UAH] = ₴")
ureg.define("THB = [currency_THB] = ฿")
ureg.define("TRY = [currency_TRY] = ₺")
ureg.define("INR = [currency_INR] = ₨ = ₹")
ureg.define("MXN = [currency_MXN] = ₱")
ureg.define("ILS = [currency_ILS] = ₪")
ureg.define("AUD = [currency_AUD] = ₳")
ureg.define("RUB = [currency_RUB] = ₽")
We could start with those and update as needed. As for base currency,
let's start with EUR and change if needed. I can fix the pint demo to adapt
;-)
—
Reply to this email directly, view it on GitHub
https://github.com/os-climate/data-platform-demo/issues/51#issuecomment-1242915324,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACHUARXTBUN7QKW3T4G2YJ3V5WJEXANCNFSM5XH4S6EQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

--
==========================
Bryon Baker
Ph.: +61(0)457531202
Email: @.***

I have this running in dev. It is publishing to the ecb-fx topic in the fx-pricing-dev cluster. It is loading all the above currencies from 1999 right now.
Sample data format is:
{"key":"EXR.D.INR.EUR.SP00.A","freq":"D","currency":"INR","currency-denom":"EUR","sender":"ECB","exr-type":"SP00","exr-suffix":"A","starttime-period":"2022-08-08T00:00:00.000+02:00","end-time-period":"2022-08-08T23:59:59.999+02:00","obs-value":81.166,"obs-status":"A","time-format":"P1D"}
{"key":"EXR.D.INR.EUR.SP00.A","freq":"D","currency":"INR","currency-denom":"EUR","sender":"ECB","exr-type":"SP00","exr-suffix":"A","starttime-period":"2022-08-09T00:00:00.000+02:00","end-time-period":"2022-08-09T23:59:59.999+02:00","obs-value":81.406,"obs-status":"A","time-format":"P1D"}
{"key":"EXR.D.INR.EUR.SP00.A","freq":"D","currency":"INR","currency-denom":"EUR","sender":"ECB","exr-type":"SP00","exr-suffix":"A","starttime-period":"2022-08-10T00:00:00.000+02:00","end-time-period":"2022-08-10T23:59:59.999+02:00","obs-value":81.468,"obs-status":"A","time-format":"P1D"}
{"key":"EXR.D.INR.EUR.SP00.A","freq":"D","currency":"INR","currency-denom":"EUR","sender":"ECB","exr-type":"SP00","exr-suffix":"A","starttime-period":"2022-08-11T00:00:00.000+02:00","end-time-period":"2022-08-11T23:59:59.999+02:00","obs-value":82.2845,"obs-status":"A","time-format":"P1D"}

@HumairAK I raise an issue #203 to get the trino connector set up. https://github.com/os-climate/os_c_data_commons/issues/203

The coding is done. Waiting on the Trino configuration

Needs the Kafka topic issue # 203 to be done.

Dependent on https://github.com/os-climate/os_c_data_commons/issues/203


Only deploy when files that need deploying are changed

Only deploy when files that need deploying are changed

.deploy noop

.deploy


Bump LiteX

In particular to grab updated terminology.

ALSO, a bump will likely fix a hang we're seeing with
--cpu-type=serv --cpu-variant=mdu

I see the same hang on an older LiteX installation (just LiteX, not CFU Playground), but not when all recent updates have been applied.

@ShvetankPrakash ^^


Responsive svg & directive

The angular directive should be resized on window changes. Find Angular function to respond to changes to client bounding box.

The way that axes are generated does not provide an easy way to respond to window changes. I am going to move on to issues with higher percieved values for now


Expose the mapped model class to the block passed to define_type

Consider ...
module CustomTypeExtension
  def extended_field(*args, &block)
    # I want to be able to know that the Index document type is mapped to the class Models::Foo
    self.underlying_model.do_something #this method doesn't really exist
    field *args, &block
  end
end

class FooIndex < Chewy::Index
  define_type Models::Foo do
    extend CustomTypeExtension
    extended_field :cat, :dog
  end
end

I'd like to be able to access the associated Model's class inside of the block passed to define_type in general, and inside of mixed in modules specifically. I went trolling through the methods but I couldn't find anything appropriate. Advice?

I'm happy to submit a PR that accomplishes this if you let me know where you'd like to see the method which exposes the model class added and what you think it should be named.

adapter.target actually

Thank you.


Tree build: filenames containing double quotes hangs and filenames containing spaces get truncated

Hello Alex,
I found these two cases in RADXIDE tree building code, I do not know if by case Alited is also affected:

filenames containing double quotes hang the application;
filenames containing spaces get truncated to the leftmost part.
Actually I'm investigating how to fix the code in RADXIDE.
I'd ask you with courtesy to let me know if by case you already solved them.
I attach the debug log of the hang for the filenames containing double quotes.
Thanks,

-Dan
debug.log

My patch attached. Actually I ignore d == "y" setting always d = " ". I 'm not sure about the cons of this couse RADXIDE does not use "y" in file names.
patch.txt

Hi Dan,
Thanks for the feedback.
Somewhere in docs I strongly recommend to avoid spaces in file names of Tcl projects, not to say about double quotes. Nonetheless such names are nicely displayed in alited's file tree, as well as unit names in the unit tree.
I don't know why your tree needs the additional actions to fix this.
Perhaps you populate the tree at creating it? I firstly create the tree and then run "after idle" with the populating procedure.

Thanks to your hints, after awaking, I produced a new patch for you encopassing the "y" to manually hide files.
Hope it is useful.
patch_last.txt

patch fixed
patch_latest.txt

It is already a 4 days that I'm successfully running patched version of RADXIDE.
Hope you can enjoy the patch for dirContent.
-Dan

Good. That's okay with my tree. I'm glad that with yours too. You might close this issue.
I should note however that some lines of your code might be simpler.
Firstly, use braces in [expr] command. This is recommended for performance. This is so in most cases: there are rather rare cases when the braces aren't used, e.g. in command substitutions.
See details in:
https://tcl.tk/man/tcl/TclCmd/expr.htm
So, istead of
expr [string length $fname]-2

use
expr {[string length $fname]-2}

Secondly, this snippet
set start [expr [string length $fname]-2]
set last [expr [string length $fname]-1]
set d [string range $fname $start $last]

could be nicely substituted with the following
set d [string range $fname end-1 end]

See https://tcl.tk/man/tcl/TclCmd/string.htm about STRING INDICES.

Thank you Alex, well received.
I wrote the last post to close the issue too... but it seems I can't do it from my side.
-Dan
Oct 12, 2024 07:11:08 Alex Plotnikov @.***>:

Good. That's okay with my tree. I'm glad that with yours too. You might close this issue.
I should note however that some lines of your code might be simpler.
Firstly, use braces in [expr] command. This is recommended for performance. This is so in most cases: there are rather rare cases when the braces aren't used, e.g. in command substitutions.
See details in:
https://tcl.tk/man/tcl/TclCmd/expr.htm
So, istead of
*expr [string length $fname]-2
*
use
*expr {[string length $fname]-2}
*
Secondly, this snippet
*set start [expr [string length $fname]-2]
set last [expr [string length $fname]-1]
set d [string range $fname $start $last]
*
could be nicely substituted with the following
*set d [string range $fname end-1 end]
*
See https://tcl.tk/man/tcl/TclCmd/string.htm about STRING INDICES.
—
Reply to this email directly, view it on GitHub[https://github.com/aplsimple/alited/issues/39#issuecomment-2408393974], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AFONMRAYEUSKKL3FLWB2WQDZ3CVOPAVCNFSM6AAAAABPQX3HROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBYGM4TGOJXGQ].
You are receiving this because you authored the thread.
[Tracking image][https://github.com/notifications/beacon/AFONMRFRG6NNDYFAM27RLGTZ3CVOPA5CNFSM6AAAAABPQX3HROWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTUPRUWPM.gif]


fCC Everything Be True Challenge --> Missing Closing Quote In Test Cases Shown

https://www.freecodecamp.org/learn/javascript-algorithms-and-data-structures/intermediate-algorithm-scripting/everything-be-true

The test cases are missing a closing quote in what is displayed in the challenge and there is no way to tell if it's the same under the hood or not - probably not because the code won't even run the way it is shown in the challenge.
truthCheck([{id: 1, data: {url: "https://freecodecamp.org", name: "freeCodeCamp}}, {id: 2, data: {url: "https://coderadio.freecodecamp.org/", name: "CodeRadio"}}, {id: null, data: {}}], "data") should return true.

truthCheck([{id: 1, data: {url: "https://freecodecamp.org", name: "freeCodeCamp}}, {id: 2, data: {url: "https://coderadio.freecodecamp.org/", name: "CodeRadio"}}, {id: null, data: {}}], "id") should return false.

The word "freeCodeCamp"  in those examples has a " in front of it but not after it - causing the code to fail.
What is the proper thing to do? Just fix it?

  **Your browser information:**

User Agent is: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0
Challenge:  Everything Be True
Link to the challenge:
https://www.freecodecamp.org/learn/javascript-algorithms-and-data-structures/intermediate-algorithm-scripting/everything-be-true

## Expected behavior

<!-- Add a clear and concise description of what you expected to happen. -->

When the test cases are used to develop a solution to the challenge the code fails. Please add a closing quote to the last two test cases shown on the challenge page so that they may be used directly in our efforts to develop a solution to the challenge.

## Screenshots

<!-- If applicable, add screenshots to help explain your problem. you can drag and drop, png, jpg, gif, etc. in this box. -->

![Screenshot 2022-03-17 at 22-28-59 freeCodeCamp org](https://user-images.githubusercontent.com/86216502/158943232-e4e707cf-524b-4b77-bce7-b41df248a3e7.png)

## System

<!-- Please complete the following information. -->

- Device: [e.g. iPhone6, Laptop]
- OS: [e.g. iOS 14, Windows 10, Ubuntu 20.4]
- Browser: [e.g. chrome, safari]
- Version: [e.g. 22]

## Additional context

<!-- Add any other context about the problem here. -->

Thanks for opening this issue.
This looks something that can be fixed by "first time" code contributors to this repository. Here are the files that you should be looking at to work on a fix:
List of files:

https://github.com/freeCodeCamp/freeCodeCamp/blob/main/curriculum/challenges/english/02-javascript-algorithms-and-data-structures/intermediate-algorithm-scripting/everything-be-true.md

Please make sure you read our guidelines for contributing, we prioritize contributors following the instructions in our guides. Join us in our chat room or the forum if you need help contributing, our moderators will guide you through this.
Sometimes we may get more than one pull requests. We typically accept the most quality contribution followed by the one that is made first.
Happy contributing.

Hi @gikf , I'm new to open source and would love to contribute to this

@anjiqbal :+1:, no need to ask for permission. Issues labelled with first timers only or help wanted are open for contributions. My previous comment includes couple of links that should help with getting started and if you'd get stuck.

Hey @anjiqbal,
It seems you have opened a pr on your own version of the repository.
Do you mind opening it here instead?
Have a nice day and happy coding 😄

Closed with #45460


ECOPROJECT-1079 Bug Fix - Update UI description

ECOPROJECT-1079 Update UI description to reflect SNR template is created automatically.

/retest

/retest

/lgtm


RenderUtil and Canvas' switch back to push()/pop() instead of shift()

And process matrices in reverse in transformPoint and flatten

Actually nevermind. Unshift is correct. A child's transforms should apply BEFORE its parent's transforms, hence unshift. The array of matrices is in transform order, not multiplication order.


improvements in the ontology viewer

A few things would be quite helpful (I'll take the example of adm:statementScore where all these are visible):

[ ] display the rdfs:comment property, it's a very important one
[ ] Rdf Type should be Type
[ ] There should be no space before columns: Rdf Type:, not Rdf Type : (only French typography uses space here)
[ ] URI is not very interesting, it could be removed (the URI already appears in the web browser)
[ ] the literals should be printed using the same format as turtle: Label: "score"@en, not Label: score
[ ] classes, individuals, etc. should also be printed using the usual format: , Range: xsd:integer not Range : integer, and there should be html links, for instance xsd:integer should be a link to https://www.w3.org/2001/XMLSchema#integer (which in this case is not interesting, but in most cases is)

Fixed as of commit d87f3cd

The links in http://purl.bdrc.io/ontology/admin/statementScore are still wrong (some links go to invalid addresses like adm:StatementAnnotationBody

Fixed as of commit 9576e72 (also added styled display of labels and comments in class view)

A space is still missing after "Comment:", otherwise it's good, thanks!

Fixed as of commit da7eaba


Online doc issue in Chrome Browser on Win 7

Chrome browser ver. 49.0.2623.108 m
I start at http://orientdb.com/docs/last/index.html and left-pane links for Introduction and sections 1-10 appear.
When I click, in the left pane, 2. Data Modeling, the left pane should expand to show items 2.1, 2.2 etc. (as it does in IE).
But instead of doing that, the Introduction, and section 2, 5, 6, 9 and 10 disappear, leaving only main items 1, 3, 4, 7 and 8.  This persists even after hitting the back button.
Not a big deal; I can read docs on IE.  But I thought you'd like to know!

I'd say the problem is somewhere else. I have the same version of Chrome on Windows7 and everything works perfectly for me (except the links in the methods, which I reported a while ago). Try emptying out your browser's cache for ODB's site. That is currently the only thing I can think of.
Scott

Thanks Scott, it worked!  For others' future reference I performed two operations (not sure which specific one worked):
I entered the URL chrome://history/ and selected and cleared all orientdb.com pages
Next I entered the URL chrome://settings/clearBrowserData and selected only the following items and cleared them from the beginning of time:
Cookies and other site and plugin data
Cached images and files
Autofill form data
Hosted app data
Content licenses
But not
Browsing history (other than orientdb.com pages)
Download history
Passwords
Also of interest:
Google's site suggests that the only site-specific removal option is for browsing history.  See
https://support.google.com/chrome/answer/95589?rd=1
Thanks again!


Rangeslider support

This is a first draft of cleaned version of rangeslider support:
Still to do:

improve UI (less clear rangeslider)
create examples
write tests
add docs
add argument to specify rangeslider layout

Add example result gifs!
See: https://github.com/predict-idlab/plotly-resampler/pull/251/commits/ad17cfd4acb499d8e48d1ecedfb90878b5c1ed11
😁

Codecov Report

Merging #254 (b23b09a) into main (574904e) will decrease coverage by 5.76%.
Report is 2 commits behind head on main.
The diff coverage is 29.89%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
@@            Coverage Diff             @@
##             main     #254      +/-   ##
==========================================
- Coverage   96.87%   91.12%   -5.76%
==========================================
  Files          14       14
  Lines        1024     1115      +91
==========================================
+ Hits          992     1016      +24
- Misses         32       99      +67

Files
Coverage Δ

...tly_resampler/figure_resampler/figure_resampler.py
59.59% <29.89%> (-28.26%)
:arrow_down:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

Ready for review @jvdd!


detect duplicate field names in payload templates resulting from substitution

Is your feature request related to a problem? Please describe.
Need a new rule to detect duplicate field names in a payload template following the rename that occurs after evaluation.
Describe the solution you'd like
Add a new validator that detects names that will collide.
invalid:
"Parameters" : {
    "static": "foo",
   "static.$": "$.path.to.value"
}

The above is valid JSON at the time of deploy but will be invalid after evaluation since static.$ is renamed to static which collides with the existing field.

Available in 3.0.1


links updated in required file

3 links are updated
Closes #9

Changes are done successfully @HassanBahati
Please review


Improve datatypes show on content assist

Should really be only additions to these two functions:
https://github.com/halcyon-tech/vscode-db2i/blob/main/src/language/providers/completionProvider.ts#L67

getParmAttributes
getColumnAttributes


photo quality issue

Hi, I encountered an issue in that my image seems to be misplayed, and the quality is bad.
This is my photo source:  https://ibb.co/3pbVDhp
and this is my output: https://ibb.co/TRDMSSS
Please let me know why, much thanks  :)

I'm pretty sure you're not actually displaying your claimed source photo. It looks like you're displaying some sort of thumbnail; in addition to the extremely low resolution, the image you're displaying in Pannellum appears to be cropped.
I can't provide any more assistance with the information you've provided. If you post your Pannellum configuration, that would be helpful.

Closing due to lack of response.


create User but  not get deliver value to insert to database

I define User model like below
import { sequelize, Sequelize } from "../db";
import { Instance, DataTypes } from "sequelize";
import {Table, Model, PrimaryKey, Column, AutoIncrement, BelongsToMany,
    DefaultScope, Scopes, CreatedAt, UpdatedAt} from "sequelize-typescript";

@Table
export class User extends Model<User> {
    @PrimaryKey
    @AutoIncrement
    @Column
    public id: number;

    @Column
    public firstName: string;

    @Column
    public lastName: string;

    @CreatedAt
    public createdAt: Date;

    @UpdatedAt
    public updatedAt: Date;
}

and Usercontroller
    public async createUser(ctx: any) {
    logger.info("Andrew - UserController.createUser()");
    const req = ctx.request.body;

    sequelize.addModels([User]);
    const user: User = new User();
    user.firstName = req.firstName;
    user.lastName = req.lastName;
    await userService.createUser(user);
    ctx.body = user;
}

UserdbService
public async createUser(user: User): Promise {
logger.info("-----------------------", JSON.stringify(user));
await User.create(user);
await User.create({firstName: "Ye", lastName: "bob"});
}
I use postman to deliver user value, but it create user without deliver value this like this, and second statament can execute. It is has value devliver from postman like below. I don't konw where is error.

Hey @chaoxihailing, the data is stored correctly, right? The only issue is, that the data does not appear in the response? Which framework are you using for the http request handling? Do you have an example repo, where I can have a look at?


[WIP] Azure EMS refresh

This PR contains the Azure refresh code with exception to the Azure model code which was previously merged in the following PR:
https://github.com/ManageIQ/manageiq/pull/3623
The Trello card that corresponds to this PR is:
https://trello.com/c/e8JwhPsu/41-3-azure-collect-inventory-for-vms-and-flavors
This PR will remain a WIP until the  Ems_Azure::verify_credentials method is implemented

Seems we need to drop the ":require => false" bit when pointing at the repo. That fixed it for me.

@djberg96 - works a charm, thanks :+1:

Overall this looks good.  A few comments above to address.
I'm not crazy about the abbreviated variable names, but I'd be fine merging for this sprint if you can go back and un-abbreviate some things in the RefreshParser.

Also, tests are passing here for Travis.  I'm guessing the WIP label should be removed?

Minor request please: assign the resource group to ems_ref so I can get at it. :)

@djberg96
I already am.  Both uid_ems and ems_ref are set to resourcegroup\name

@blomquisg -
yes, I can review the variable names later this week.
I need to implement the vertifiy_credentials method, then i can remove the WIP label.

@bronaghs Oops, my bad!

@bronaghs @blomquisg @Fryguy what is the status of this PR that we were trying to merge in Sprint 28?

@chessbyte - Did not make it into Sprint 28 due to insufficient specs and because of a spec test failure resolved by a refactor of the arm rest gem: https://github.com/ManageIQ/azure-armrest/pull/56
I expect to have a complete PR tomorrow.

@bronaghs We have specs that are passing Travis.  Is this ready for final review?

@chessbyte - yes it is.


loadGltfUris can run out of file handles

loadGltfUris currently runs out of file handles for models that reference lots of external resources. I have a model that references over 9000 images, and it fails with:
Unhandled rejection Error: EMFILE: too many open files

Hmm. looks like this is because loadGltfUris uses fs instead of fs-extra. fs-extra under the hood wraps graceful-fs, which avoids EMFILE exceptions by queuing file requests or something.


got an error when using my own dataset

Hi Benedikt
Thank you so much for this work, I have achieved training and prediction with the help of your documentation
But I encountered difficulties when I wanted to use it on non-lidar point cloud data. This data comes from my depth camera [Bumblebeexb3]. The following is a sketch of it opened in Cloudcompare and in txt format.

The data contained in it is x y z intensity.It may not be obvious but it is a landslide of an indoor experiment, and I hope to predict the landslide through this work.
My problem is that he reported this error when I converted the txt file into a .bin file through the code.

I use numpy to convert the data
point_cloud = numpy.loadtxt("try.txt") a.tofile("try.bin")
When I use Cloudcompare to convert the txt file into .pcd and open it to add the intensity header file and then convert .pcd into .bin through another script, I can run through the above preprocessing link, but Dataloader does not read the data

You can also see that the calculated averages are all 0.
Could you please give me some suggestion or solution？

Hey @xutianq999!
The error you get during preprocessing is because we assume to have 3D points in the local frame of a rotating laser scanner with the field of view defined in parameters.yaml. During preprocessing, the point cloud is represented as a range image using a spherical projection.
You could try to process the disparity image of your stereo camera instead of the resulting 3D points. Given the baseline and the focal length of your setup, you can compute the depth values resulting in a 2D depth image. It might be easier to write your own dataloader. Just make sure it returns past and future images and the required meta data. The images should be torch.tensor objects of size (n_channels, n_steps, height, width), where the channel could be the disparity/depth in your case.
Also, you should disable the CIRCULAR_PADDING since this only holds for images with 360 degrees horizontal field of view.
Best
Benedikt

Thank you very much. I have been troubled by this problem for a long time and tried many ways. I did not expect you to reply so quickly. Once again, I would like to express my high respect to you.
Regarding the suggestion you mentioned, when I was reading the code, I noticed the part that forms the range_iamge by 3D point projection. I also tried to change some properties in the yaml file but did not achieve good results. I noticed that you also mentioned in your paper that CIRCULAR_PADDING will improve the data of the mechanic lidar to a certain extent.
Please forgive me and confirm to you again, I should rewrite Dataloader, not to read 3D point cloud files like .bin files, but to directly input depth maps, right? As the landslide data I provided above, can I convert it into a depth map, and then directly use such data for training?
Thank you again for your reply, you have given me great encouragement and courage

You are welcome! Yes, the circular pattern helps to propagate features from one side of the range image to the other one. This is important since the rotating LiDAR covers the whole 360 degrees in the azimuthal direction, which means that the left and right range image boundaries are actually connected. Since this does not hold for your depth image, you should disable the circular padding.
Regarding your problem: Yes, instead of reading and projecting 3D point clouds, you can directly load the depth maps in the dataloader. Just modify the __getitem__ as mentioned above. Also, the MIN_RANGE and MAX_RANGE in the parameters.yml are used as limits for the prediction. If you want to predict the depth, these should be the minimum and maximum depth you want to predict. You also need to change some parameters like the HEIGHT and WIDTH of your depth maps.
For now, I would train the model without using the Chamfer Distance loss and only use the range/depth L1 loss and the valid mask loss. If you want to use Chamfer Distance loss as well, you need to modify the get_target_mask_from_range_view method to get the 3D point cloud from your predicted depth map.
So, there are some modifications needed to make this work with depth maps, but it should be possible. I actually used this codebase and modified it for the Waymo Occupancy and Flow Prediction Challenge which worked quite well.
Best regards
Benedikt

Hey,

Yes, indeed. You just have to modify the dataloader to read your depth maps. I did not try to train it on depth maps, but I don't see a reason why it should not work.
If you have pixels without/with invalid depth information, the mask should be zero, and therefore ignore these. This happens for example for the sky.
If you compute the 3D point, this should be possible. Again, you have to modify the code for this.

Best
Benedikt

Hey,

Yes, indeed. You just have to modify the dataloader to read your depth maps. I did not try to train it on depth maps, but I don't see a reason why it should not work.
If you have pixels without/with invalid depth information, the mask should be zero, and therefore ignore these. This happens for example for the sky.
If you compute the 3D point, this should be possible. Again, you have to modify the code for this.

Best Benedikt

Thank you for your quick replies one by one.
I will try to modify the code and wish you a Merry Christmas and good scientific research!
Best
yzy


[Feature Request] Android Wear support

With more competent Android Wear devices popping up on the market and 3.0 around the corner, Android Wear support would allow me to finally leave behind the most important use for my Pebble 2. For me, something as simple as a start/stop button and notifications would be sufficient, but I'm not sure how strange my use case is.
I'm not experienced with Android Wear development, but if someone wants to point me in the right direction I can take a look at the code and see whether I can figure out where to start on such an implementation. Thanks!

At the moment you can start a session from the phone app and then see the countdown and control the next sessions though the notification buttons on the wearable. I assume you would like an Android Wear standalone app which I assume would not be trivial to implement.

Unfortunately for me, the notification doesn't seem to make it to wear os since it is a silent notification. A wear os companion app would be amazing as it allows me to put my phone away to avoid distraction during a session. It would likely not be trivial, but likely would not be a huge undertaking. For me, the lack of a watch support has pushed me to excercise timer for now even though this app is an overall better experience.

It doesn't make it to Wear OS for me either, but I believe it's actually because it's a persistent notification. Persistent notifications on the phone aren't synced to Wear OS AFAICT, and there's no way to change that.
Of course, a Persistent notification is exactly the right thing for the app to be using, so making it not Persistent isn't a good solution.
I am pretty sure there is a flag in android notifications specifically for whether a notification should display on Wear devices,  so I'd look there first.


Setting Flutter Architecture

Currently no initial Setup for Flutter Project.
to set up and add in a folder called frontend.

please i would love to work on this, assign it to me, thanks.

@fz3hra don't forget to add the architecture for bloc, thanks


Add 45-rule implementation

Create an ojAlgo implementation that uses 45 rule.
If every row/column/square sums to 45, this solves the Sudoku.
I don't know if it's worth creating a from-scratch branch-and-bound implementation using 45-rule as well.

Decided I'm not going to implement this. The generalized approaches already implemented are translatable to more real-world examples like scheduling.


fix #70 delete the hack method disable io for http2's read loop

Codecov Report

Merging #90 into master will increase coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master      #90      +/-   ##
==========================================
+ Coverage   22.31%   22.32%   +<.01%
==========================================
  Files          40       40
  Lines        3500     3499       -1
==========================================
  Hits          781      781
+ Misses       2683     2682       -1
  Partials       36       36

Impacted Files
Coverage Δ

pkg/config/parser.go
3.23% <ø> (ø)
:arrow_up:

pkg/config/config.go
12% <ø> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 091cf92...5711fa9. Read the comment docs.


validate the kube-dns is nil AND add ignore rules from coredns

exclude the services that is kube-dns or coredns

Currently, edgemesh hijacks all services on the cluster ip CIDR. So what I consider is to filter the services that edgemesh does not want to hijack.
Maybe we can add a label to the service that we don't want to hijack, and then process the label in the code. However, I think this(let users add labels) will make users feel a little troublesome.

Currently, edgemesh hijacks all services on the cluster ip CIDR. So what I consider is to filter the services that edgemesh does not want to hijack.
Maybe we can add a label to the service that we don't want to hijack, and then process the label in the code. However, I think this(let users add labels) will make users feel a little troublesome.

It seems that we can use the filtering feature of label in this way, but this requires adding an agreed label.
	// Other services we want to ignore...
	otherIgnoreServiceList, err := proxier.kubeClient.CoreV1().Services("*").List(context.Background(), metav1.ListOptions{LabelSelector: "proxyby!=edgemesh"})
	if err == nil && otherIgnoreServiceList != nil && len(otherIgnoreServiceList.Items) > 0 {
		for _, item := range otherIgnoreServiceList.Items {
			klog.V(4).Infof("ignored not containing proxyby=edgemesh label service: %s", item.Name)
			ignoreRulesIptablesEnsureMap[strings.Join([]string{item.Namespace, item.Name}, ".")] = &item
		}
	}

/lgtm

It seems that we can use the filtering feature of label in this way, but this requires adding an agreed label.

Right, there is no problem, we can do it.
However, if some users only use edgemesh at the edge node (without deploying kube-proxy), it means that we need to manually add a agreed label to each service.

Can I add this code after this pr?

Can I add this code after this pr?

Yes, but I suggest you raise an issue first, describing clearly why you want to do this. Maybe others need to understand the background and your intentions.

Okay, let me test it in my own environment first!

Nice~

/assign @fisherxu


Build (in apk) Size (0.0, 0.0) is the result.

When we build the application as apk, I get Size (0.0, 0.0). It works in a test environment. But the application does not work when we build it as apk. What is the reason of this?

@sbilketay are you still having this problem? I need to know if I should use this on my app

Can you provide more information? Are you saying that you are using the Dart Size class in your code an it has an unexpected value in certain situations? Sounds like there might be a difference between your test/production environments or debug/release builds. Does the problem go away when you remove this package?


Example for zip

The Python build-in function zip is very useful for going throug two folders in parallel. In image analysis, this is often used to go through a folder of images, and a corresponding folder of label mask images.
We can use these folders as examples:

data/BBBC007/images
data/BBBC007/masks

We should write a new notebook docs/33_batch_processing/16_zip_folders.ipynb that demonstrates how to do this. In this notebook, the zip-function is used to open both folders mentioned above and goes through the image files in that folder using zip. BOth images, the intensity image and the label mask images are opened in the for-loop and the average intensity of all labeled objects, plus the number of objects are stored in a dataframe. After the for-loop, the dataframe is shown.
git-bob think about this.

git-bob think about this.

Sounds great! We should also add the notebook to the _toc.yml file.
git-bob implement this!


python311Packages.stripe: 5.2.0 -> 5.4.0

Changelog: https://github.com/stripe/stripe-python/blob/v5.4.0/CHANGELOG.md
Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 227755 run on x86_64-linux 1

8 packages built:

gigalixir
gigalixir.dist
python310Packages.stripe
python310Packages.stripe.dist
python311Packages.stripe
python311Packages.stripe.dist
sourcehut.metasrht
sourcehut.metasrht.dist

Result of nixpkgs-review pr 227755 run on x86_64-linux 1

8 packages built:

gigalixir
gigalixir.dist
python310Packages.stripe
python310Packages.stripe.dist
python311Packages.stripe
python311Packages.stripe.dist
sourcehut.metasrht
sourcehut.metasrht.dist


Is operator not designed for emqx broker v5 ?

Describe the bug
When I try to deploy a v5.0.6 emqx image the resulting statefulset does not properly create emqx cluster.
It is like the operator v1.2.5 is not designed to operate emqx v5 broker images. So my question: is this a bug or it is just not designed for v5 yet ?
To Reproduce
deploy something like:
apiVersion: apps.emqx.io/v1beta3
kind: EmqxBroker
metadata:
  name: emqx
  namespace: emqx-operator-system
spec:
  replicas: 3
  emqxTemplate:
    image: emqx/emqx:5.0.6
    serviceTemplate:
      metadata:
        name: emqx
        labels:
          app: emqx
      spec:
        selector:
          app: emqx
        type: ClusterIP
        ports:
          - name: mqtt
            port: 1883
            protocol: TCP
            targetPort: 1883
          - name: mqttssl
            port: 8883
            protocol: TCP
            targetPort: 8883
          - name: dashboard
            port: 18083
            protocol: TCP
            targetPort: 18083
    config:
      prometheus.enable: "true"
    modules:
      - name: emqx_mod_trace
        enable: true
  persistent:
    resources:
      requests:
        storage: 2Gi
    accessModes:
      - ReadWriteOnce

After the broker starts I have some warnings in logs and autocluster is not working. Every broker node has a cluster size of 1 -> they don't form a cluster.
emqx [warning] unknown_env_vars: ["EMQX_CLUSTER__DISCOVERY","EMQX_CLUSTER__DNS__APP",
emqx                    "EMQX_CLUSTER__DNS__TYPE",
emqx                    "EMQX_DASHBOARD__DEFAULT_USER__LOGIN",
emqx                    "EMQX_DASHBOARD__DEFAULT_USER__PASSWORD",
emqx                    "EMQX_PLUGINS__ETC_DIR","EMQX_PLUGINS__LOADED_FILE"]
emqx prometheus.enable = EMQX_PROMETHEUS__ENABLE = true
emqx rpc.port_discovery = EMQX_RPC__PORT_DISCOVERY = manual
emqx log.file_handlers.default.enable = EMQX_LOG__FILE_HANDLERS__DEFAULT__ENABLE = false
emqx log.console_handler.enable = EMQX_LOG__CONSOLE_HANDLER__ENABLE = true
emqx cluster.dns.name = EMQX_CLUSTER__DNS__NAME = emqx-headless.emqx-operator-system.svc.cluster.local
emqx node.name = EMQX_NODE__NAME =<EMAIL_ADDRESS>
emqx 2022-08-28T10:28:07.426244+00:00 [warning] unknown_env_vars: ["EMQX_CLUSTER__DISCOVERY","EMQX_CLUSTER__DNS__APP", "EMQX_CLUSTER__DNS__TYPE", "EMQX_DASHBOARD__D
emqx Listener ssl:default on :8883 started.
emqx Listener tcp:default on :1883 started.
emqx Listener ws:default on :8083 started.
emqx Listener wss:default on :8084 started.
emqx Listener http:dashboard on :18083 started.
emqx EMQX 5.0.6 is running now!

Expected behavior
The broker starts and the number of nodes in the cluster via emqx dashboard is 3.
Anything else we need to know?:
It's like the operator v1.2.5 is not configuring proper env variables for broker v5. That's why logs show some warnings.
Environment details::

Kubernetes version: 1.21.10
Cloud-provider/provisioner: self-hosted
emqx-operator version: 1.2.5
Install method: helm

Hi，@mariusstaicu I'm sorry, the apps.emqx.io/v1beta3 is not designed to operate EMQX v5 broker, We plan to release the new APIVersion Kind tomorrowIt: apps.emqx.io/v2alpha1, Kind: EMQX, and it is specifically designed for EMQX v5

Hi @mariusstaicu EMQX Operator 2.0.0 Released, please see https://github.com/emqx/emqx-operator/blob/main/docs/en_US/getting-started/getting-started.md#deploy-emqx-5

thanks!


HSE Electronic Structure K-Points failure

Describe the bug
Using electronic-structure.vasp.matproj-hse-full:
Single point energy calculation finishes without an issue, but when trying to do band structure or density of states the calculation fails due to the k-points not being compatible with HF.
To Reproduce
Location on Warwulf:
/media/synology/user/becca/electrolytes/Mg_structures/MgSnF6/simmate/hse_band
Error
error in IBZKPT_HF: two k-points are equivalent           1          60
this will cause problems in the HF routine
Versions

Simmate version: 0.12

Additional details
No response

pymatgen recently fixed this, so I can use their code to help (see here)


Question: Possibility to scroll to first occurrence?

Hi there, I've come across your article regarding the CSS Highlighting API and think it is a great approach to text highlighting. I have a use case where the text that is highlighted is potentially further down the page, so user won't notice it without scrolling down themselves.
Is there a way to scroll to the first (or any) of the highlighted texts? I have had a look at the source code, but am not sure how the ranges exactly work, or if they could be used for such a feature.
Thoroughly enjoyed your article, great work!

Thanks for your feedback!
There is no built-in way to scroll to the first occurrence. You'll have to fork the lib to add this capability.
We won't implement that feature in the library because it was designed for filter-as-you-type inputs. Scrolling to the first result may hide the input, even though the user hasn't finished typing their search term. This is a bad UI.

Thanks for your answer! I didn't expect that you would add this feature to the library as is, as the use case I have is quite different (only highlight one longer text)—it was more a general question. Though reading through the code again, I think I know how I can proceed.


Nomination Green Reviews WG Tech Lead: Ross Fairbanks

Dear TAG ENV and Green Reviews WG members,
The Green Reviews WG has an open need for a Tech Lead (TL) who will help lead the design and implementation of the sustainability assessment pipeline and its various components. It is my pleasure to nominate Ross Fairbanks (@rossf7) for the TL role for the Green Reviews WG.
Ross has been an invaluable member of TAG ENV and throughout the bootstrapping process of the Green Reviews WG. He has spearheaded the design and implementation of the pipeline through his active technical contribution to the infrastructure, cluster, and application provisioning, among other things. He has been stewarding the project's technical work through issues and PRs while leading the open-source community discussions in the WG's various spaces, including the WG and TAG repositories, TAG & WG meetings, design document, etc.
In the broader open-source cloud-native ecosystem, Ross has contributed to the implementation and advocacy through talks and blog posts for tooling used to gather proxy metrics for environmental sustainability. He contributed code to Scaphandre's Kubernetes integration, led discussions about Kepler, and authored a blog post about KEDA's carbon-aware scheduling feature, among other things.
Please consider this nomination for Ross. His expertise and wealth of knowledge will be of tremendous help to continue the project forward as we continue to develop the pipeline, implement its various components, and continue our collaboration with CNCF project maintainers.
All community members are welcome to vote by leaving a comment on this issue. Votes are non-binding for community members and binding for TAG Chairs & TLs.
As mentioned in the TAG ENV Governance:

Community nominations will run for at least 2 weeks, with a maximum up to 1 months. The TAG Chairs will take a vote. Upon close of nominations, a decision should be made within the next month.

Regards,
Niki

I vote for @rossf7!

NB vote for Ross!

+1 binding

A vote for @rossf7!

+1 B — thanks for all the work! 🙌

+1 NB

+1 Binding.

The nomination is now open for over two weeks and passed the vote. I opened the PR to add @rossf7 as a TL to WG Green Reviews. Thank you very much for your volunteering and congratulations!


test: more benchmarking for honk and plonk

Please provide a paragraph or two giving a summary of the change, including relevant motivation and context.
Checklist:
Remove the checklist to signal you've completed it. Enable auto-merge if the PR is ready to merge.

[ ] I have reviewed my diff in github, line by line and removed unexpected formatting changes, testing logs, or commented-out code.
[ ] Every change is related to the PR description.
[ ] I have linked this pull request to relevant issues (if any exist).

Benchmarking Plonk vs Honk
Added parallelization of add_scaled function in polynomial.cpp

closed because this was a testing branch


AudioResyncThreshold value should be persisted

Always at 0.02 after re-start of Tooll, despite of changing it in the UI (and saving)
Changing it in the json file works.

Fixed.
It's interesting, however, that you have to adjust this value. What's you current screen refresh rate and what threshold do you have to set it to?


Utility class enhancements - Refine documentation

Examine documentation, suggest ways to enhance...
Relates to:
#397
#507
#509

@mattnolting do you want to keep this one open?


fix(e2e): relative imports

v1.1.2 has been released and contains a fix for the issue. Feel free to reopen the issue or to submit a new one if you meet any problems.