[1pt] PR: Tool to compare synthetic rating curve with benchmark rating curve (sierra test)

Create tool to compare synthetic rating curve with benchmark rating curve (sierra test). resolves issue #293; resolves issue #308
Additions

usgs_gage_crosswalk.py: generates usgs_elev_table.csv in run_by_unit.py with elevation and additional attributes at USGS gages.

Data columns: ['location_id','HydroID','dem_elevation','dem_adj_elevation','min_thal_elev', 'med_thal_elev','max_thal_elev','str_order']

USGS gage is moved to closest FIM stream segment using linear referencing and elevation data is collected at the adjusted point location.

rating_curve_comparison.py: post-processing script to plot and calculate metrics between synthetic rating curves and USGS rating curve data.

Metrics: NRMSE, mean_abs_y_diff, percent_bias

Statistics are calculated at the 1.5, 5, and 10 year NWM recurrence interval as well as USGS observed flows. The group argument can be used to calculate statistics on a slice the data.

example call:
python3 /foss_fim/tools/rating_curve_comparison.py -output_dir 'data/outputs/dev-usgs-crosswalk_new_fr_c' -gages 'data/temp/tsg/usgs_rating_curve/usgs_rating_curves.csv' -flows '/data/inundation_review/inundation_nwm_recurr/nwm_recurr_flow_data' -j 14 -group 'recurr_interval'

Changes

update aggregate_fim_outputs.py call argument in fim_run.sh from 4 jobs to 6 jobs (optimizing API performance)
reroutes median elevation data from add_crosswalk.py and rem.py to new file (depreciating hand_ref_elev_table.csv)
adding new files to viz_whitelist in output_cleanup.py

Todos

add function to get USGS rating curve data from WRDS API instead of csv

Note
This feature branch will have no affect on the performance of FIM.

I was able to verify that for the 1.5 year recurrance interval for HUC 01010003 (gage 01013500) that the correct nwm return flow was being used and that the correct elevations were interpolated for the recurrance flow (also verifying units). I also spot-checked the snapping of the USGS gage to the stream network and was able to replicate results from usgs_elev_table.csv (did not attempt to calculate median thalweg elevation).  I am getting some discrepancies with the statistics when I try to calculate by hand. I am currently getting a 2nd opinion to make sure my math is correct.
Otherwise some (mostly minor) things that may be helpful:
-Appear to be missing leading zeros for USGS gage sites. The leading zeros are present in the usgs_elev_table however they are not present in the nwm_recurrence_flow_elevations.csv and the usgs_interpolated_elevation_stats.csv
-Can we truncate elevations/flows to the nearest hundredth when creating csv files
-Possible to untether the output_dir argument from the fim_run output directory, maybe add another argument to place output?
-Can we add the NWM feature_id to the usgs_elev_table? This will make it a lot less hassle to walk through all the intermediate steps if we need to.
-How are the aggregate scores calculated? Are we averaging all the individual metrics from usgs_interpolated_elevation_stats.csv (NRMSE, Percent Bias, MAD), or are we calculating actual metrics but on a much large sample size?
-Can we reference units wherever possible otherwise this may lead to some confusion ( mean_abs_y_diff to mean_absolute_elev_diff_ft , elevation to elevation_<units>).
-Make all columns lower case (NRMSE to nrmse)
-Multiple hucs in the /data/outputs/dev-usgs-crosswalk_new_ms_c/ directory appear to have all of the fim_run output (like they crashed?)
-I think we could probably just produce the FIM-USGS_rating_curve_comparison.png plot as it appears to be the original input data (lets consult with Ryan on this). The legend appears to be quite a distance from the actual plot. If this is time consuming, we can tackle this on another feature branch. I wonder if we can use consistent snake case in the plot name.
Nice job! This is going to be a huge help in assessing rating curves. If you have questions please reach out.

I am running this command:
python3 /foss_fim/tools/rating_curve_comparison.py -fim_dir /data/outputs/dev-usgs-crosswalk_PR_ms_c -output_dir /data/temp/tsg/ba_review -gages /data/temp/tsg/usgs_rating_curve/usgs_rating_curves.csv -flows /data/inundation_review/inundation_nwm_recurr/nwm_recurr_flow_data -group 'recurr_interval' -j 6
And I am getting this message:
Traceback (most recent call last):
  File "/foss_fim/tools/rating_curve_comparison.py", line 371, in <module>
    log_file = open(join(output_dir,'rating_curve_comparison.log'),"w")
FileNotFoundError: [Errno 2] No such file or directory: '/data/temp/tsg/ba_review/rating_curve_comparison.log'

@TrevorGrout-NOAA I just ran the same command with no issue. Maybe check your permissions?

I am able to get this to run.
I am still having issues replicating statistics and I would like to replicate at least one site manually. For example in HUC 02040105, location ID 01446500, we have the following metrics:

location_id
feature_id
recurr_interval
discharge_cfs
HUC
HUC4
str_order
source
elevation_ft

1446500
2588381
1.5
32519.19
2040105
204
6
USGS
236.06

1446500
2588381
5
63891.71
2040105
204
6
USGS
240.07

1446500
2588381
10
93429.28
2040105
204
6
USGS
243.1

1446500
2588381
1.5
32519.19
2040105
204
6
FIM
259.57

1446500
2588381
5
63891.71
2040105
204
6
FIM
267.72

1446500
2588381
10
93429.28
2040105
204
6
FIM
273.98

I am manually getting the following metrics:
nrmse = 3.91, mean_absolute_difference_ft = 27.35, Percent Bias = 11.41
Here is what is reported:

location_id
nrmse
mean_abs_y_diff_ft
percent_bias

1446500
1.054508
0.009878
13.19304

After running the latest commit, I was able to successfully run and replicate statistics for HUC 02040105 Site 01446500 and manually replicated the output stats.  I also verified the aggregate scores for the 1.5 year return interval. The suggested edits were incorporated. This is pretty impressive. Good Job!
New stats are shown:

location_id
nrmse
mean_abs_y_diff_ft
percent_bias

1446500
3.908029
27.34667
11.40664

@BrianAvant-NOAA Please make sure this feature branch is fully up to date with dev prior to the merge.

@BrianAvant-NOAA Make sure to update the changelog also.


Mismatched constraints between backend and frontend, for listings views

The listing importer scripts uploaded unit/property/listing data that the frontend wasn't able to display (because the frontend expected the data to conform to certain requirements). We should either tighten the constraints on what the backend will accept, or make the frontend resilient to the data accepted by the backend (or, probably, some combo of the two). As we decide whether to change the frontend vs. backend, we could use the HRD data as a guide.
First step: list the ways the frontend broke because of unmet data-format expectations.
Done criteria: we never get a frontend error for data that was accepted by the backend. To narrow the scope a bit, let's focus this issue on the /listings and /listings/{id} views.

One thing to also keep in mind: does this affect the validation in the forms for editing these in the property manager portal? I wouldn't want us to break that validation in the forms for required fields.

I futzed with minimal-listing.json for a while and ultimately wasn't able to have anything accepted by the backend that broke the frontend (note: I only looked at the listings overview and listings detail pages). Maybe our/Exygy's changes fixed the gaps I saw when I first was uploading listings?
What I did: created https://github.com/CityOfDetroit/bloom/pull/288 to trim down minimal-listing.json so that it is the true "minimal listing" (i.e. it only includes fields that are required by the backend).
Two possible next steps:

Import this new minimal-listing.json and make sure that the property manager portal doesn't break? (per Will's comments) If the property manager portal does break, then we need to either remove make the partner portal behave more gracefully in the absence of certain fields, or tighten the constraints on the backend.

Note: I think to really test this, we'll want to associate the "Minimal Listing" with a certain property manager, and then log in as that property manager. This may take a little futzing on the backend or in the DB.

Look at the listings overview page and the listings detail page and figure out which UI components need to be changed to more gracefully handle the absence of certain fields. Or: for data that we absolutely do want, add constraints to the backend.

The "done criteria" from the first comment are met, and the work to "more gracefully handle the absence of certain data" is tracked in https://github.com/CityOfDetroit/bloom/issues/228 - closing this.

Hm, in the process of working on #341, I believe that I supplied all the information in the seed listings needed (from looking at minimal listings), but I'm getting an error when I try to load the individual unit page: Cannot read property length of null.

Just re-opening to track this.

Ah, I discovered that when you specify a reservedCommunityType, you must also specify a reservedCommunityDescription (even though it looks like there are some duplicate fields inside reservedCommunityType)


Install rust toolchain

before building & testing the extractor

It seems to work fine for me!
/lgtm


Sep 14 - Azure DevOps

Event Information

Category
Details

Reactor Topic
<AI Services - Coding, Languages, and Frameworks - DevOps and Dev Tools - IoT - Mixed Reality - Cloud Development - Data Science and Machine Learning - Emerging Technologies - Personal and Business Development>

Monthly Topic (if applicable)
Skill Up September

Weekly Topic (if applicable)
DevOps

Stream Topic
<2-5 words describing the topic>

Stream Date
September 14, 2020

Stream Time
4pm PT

Stream Duration
60 minutes

Title
<150 characters or less >

Description
<500 characters or less>

Speaker

Moderator

SharePoint ID
7857

Additional Resources:

Event Checklist
10 Weeks Prior

[ ] Reactor Content Team (RCT) or Speaker opens a new issue for an upcoming stream filling in the Event Information table. Both RCT and Speaker are marked as assignees.
[ ] RCT and Speaker discuss details of the event in the comments of this issue. RCT adds a confirmed-event label and global-twitch-event tag to the issue and adds event to the Global Live Streams project. Issue will appear in the Upcoming Streams column
[ ] Speaker adds a ready-for-marketing-edit label so the RCT can review title and description

8 Weeks Prior

[ ] RCT does a review of title and description and removes the ready-for-marketing-edit label
[ ] RCT moves the issue to the Ready for SharePoint column
[ ] RCT informs the Reactor Marketing Team of the new event information
[ ] RCT provides speaker with assets (e.g. slides, overlays) and the survey data for the event
[ ] RCT moves the issue to the MONTH Streams Prep column (according to the month it will be streamed in)

2 Weeks Prior (Done in /microsoft/Reactors)

[ ] Speaker opens a PR with the new content for the stream on the customer-facing Reactor repository and adds the content-development and global-twitch-event tag to the PR.
[ ] Speaker adds to the same PR a modification to the Streaming README to include their stream information.
[ ] Speaker adds the RCT assigned to the issue as the reviewer when it is ready for review/merge
[ ] RCT reviews, approves, and merges the changes

1 Week Prior

[ ] Speaker and Moderator test setup for streaming (instructions coming soon)
[ ] RCT checks in with the speaker and moves the issue to the Ready to Stream column

During and After Stream

[ ] Speaker and Moderator run the stream
[ ] RCT submits recording to be uploaded to the Reactor YouTube and to be made available on LearnTV
[ ] RCT moves issue to the Stream Complete column

I think it would be a good idea to talk about Azure DevOps tools and services:
https://azure.microsoft.com/en-us/solutions/devops/?2000709=&OCID=AID2000709_SEM_XqjyCAAAAE3ddTAk:20200623023617:s&msclkid=deea50c4fe3517c142190e354eeb5259&ef_id=XqjyCAAAAE3ddTAk:20200623023617:s#tools

@KimberlyBolton1 - ready for edit!

@sguthals I love your title! I decided to riff off your theme, let me know your thoughts:
Azure Pipelines, and Boards, and Monitors—Oh My!
With Azure, you have access to a variety of DevOps services that help you plan smarter, collaborate better, and ship faster. Among the most popular are Azure Pipelines, Azure Boards, and Azure Monitor. During this session, a “Wizard of Azure” will dig into these tools and show how you can customize them to create your own personal workflows. It’s time you followed the yellow brick road to better DevOps!

I worry a little about the "Wizard of Azure" only because it will probably be me and I'm still new :P maybe..."explorer of Azure" or something?

@sguthals Perhaps you prefer "Auntie Em" or "Toto"? :) JK--how about this then:
Azure Pipelines, and Boards, and Monitors—Oh My!
With Azure, you have access to a variety of DevOps services that help you plan smarter, collaborate better, and ship faster. Among the most popular are Azure Pipelines, Azure Boards, and Azure Monitor. During this session, you will dig into these tools and learn how you can customize them to create your own personal workflows. Be brave, have heart, and get your brain ready because it's time follow the yellow brick road to better DevOps!

@KimberlyBolton1
Love it!
🚢

@mebaumb abstract final and moved to Ready for Sharepoint.

Event published on SharePoint, added to CH9 calendar, and stream reminder email sent. (7/20)

Event is cancelled


Switch cluster on dashboard

In beta it´s not possible anymore to switch a cluster above another existing cluster. So I can not adopt the graph.

Release 0.4.3 gonna fix this


Error while decrypting the keys

I am able to encrypt the key. While decrypting the key it is throwing JSON parse error unexpected json error. Please help me out.

Could you share some more data, logs, exceptions, etc?

From the ebics I am getting the response as "SUBSCRIBER State in invalid" after initializing the ini. Is it an issue with the keys?.
I have seen one of the comment  that ebics generates 3 different RSA keys to initialize the ini,hia and HPB. I have compared the keystring generated by the ebics to initialize ini,hia both are same.
In one of the comment I saw that we can download ini letter for that we have to give path of ini.hbs file in my case I am not getting the hbs file.How can I create this.


[EMM-1882] AppDownloadURLHost config is not used as App download URL

Reference: https://wso2.org/jira/browse/EMM-1882

Tested on the IoT 3.1.0 cluster and issue did not occur, hence resolving the issue

Tested the issue on IoT Server 3.1.0 cluster and issue did not occur, hence resolving the issue.


Prep ingestion Makefile

Instead of putting all ingestion related work in the root project Makefile
We can then import the ingestion specific make commands into the project Makefile

Hi i would like to work on this.

Hey @harshach , I would love to work on this issue. How can I get started?

hi @RahulSurwade08, assigned the issue to you. Thanks for pinging!
We have a few recipes in the Makefile that can be moved out from the root project and brought within the ingestion/ directory:

env38
clean_env37
install
devinstall
install_apis
install_tests
install_dev
install_all
precommit_install
lint
py_format
py_format_check
generate
devgenerate
run_ometa_integration_tests
unit_ingestion
run_python_tests
coverage

@rahulsurwade any updates here? Let us know if you need any help


Trin-Bridge workspace

What was wrong?
Extracts bridge functionality from the bridge branch into a encapsulated trin-bridge workspace that emulates the functionality of ethportal-bridge.
How was it fixed?
Add support for 3 bridge modes.

--mode=latest
--mode=backfill
--mode=u64 (backfill from epoch # u64.

To-Do

[ ] Add entry to the release notes (may forgo for trivial changes)
[ ] Clean up commit history

blocked by #615 & #613, imo it's better to get those across the line before turning attention to this pr

Applying this change should fix #613, right? Not sure what else there is to do with 613, then.
Maybe the best thing to do is to apply the commit from 613 to this PR.

@carver "Applying this change should.." - what change exactly are you referring to? Imo, we just close #613 if/when we merge #615. Then I'll rebase this over #615 and then we should be operating in bridge mode with valid bodies/receipts

@carver Thanks for the review! It was for sure a hefty pr. I agree that it's still early in the trin-bridge march towards stability, and there are undoubtedly more changes to come to this workspace. But, also the priority is to get this behemoth merged, and then future issues can be addressed in smaller chunks. I addressed every comment here, and there are only a couple where I left a response for you to :eye: - but even those I'm fairly confident can be considered "resolved". Of course, if you feel otherwise, I'll address all suggestions in a future pr. Tracking the only "major" lingering issue in #642


Added a staging environment

Staging is a way to run production locally - Kubernetes and all. A lot has to go into the deployment process. Even more has to change with the images designed for production. This PR is a work in progres, so watch this space. When this done, it will close issue #72

Too many file changes, trash and restart please

Is there anyone you think we could ask about loading (and reloading) Cassandra schema changes? Could ask around RC?

I'm going to leave this branch up but I am going to close it since we wont be merging this, it's only for archive purposes.


Can't run ./eosd

./eosd used to work
config.ini seems to be fine
enable-stale-production = true

do you have the genesis.json put at the right place?

@guanqun
The problem was in trying to run ./eosd after after being suspended(ctrl + z).
But daemon was still running.
Fixed that by using fg command.
The problem is over. Thank your for feedback.

@ioleksiu I'm facing same issue sir, please help me out with the fg command syntax!

hey @ioleksiu i want to know about your error. Is your error fixed?

@rupaliathashcode try with ./eosd --resync

Thank you so much sir @jatinmehta420


iOS DynamicType fonts

We need to match system font (especially size) on iOS devices
https://www.webkit.org/blog/3709/using-the-system-font-in-web-content/

And OS X now support it too - https://developer.apple.com/library/prerelease/mac/releasenotes/General/WhatsNewInSafari/Articles/Safari_9.html


ES5 support

I am a big fan of fantasy-land. Today I come across an issue when trying to use fantasy-land laws to test my reference implementation of Identity type. My tests run under node 4, 5, 6, 7. The codebase of fantasy-land will run only for node of version => 6.  When I want to use the laws on node 4 or 5 I got syntax error due to the fact that there are places where  object destructuring are used. And destructuring is only supported on node => 6.
I propose to create a prepublishOnly entry in package.json, and use babel to transpile the sourcecode into valid ES5 code. That way even older nodes can benefit from fantasy-land. Before publising to npm, ES5 tranpiled version of code will be created in ./lib directory and 'main' entry in package.json will point to this dir. Take a look here what I have in mind.
I can create a PR with all described above, I just need a green light to do so.
Thank you

@davidchambers is this issue solved with https://github.com/fantasyland/fantasy-laws/pull/1 ?

We're currently using => syntax in fantasyland/fantasy-laws#1, but we can easily switch back to ES5-compatible syntax.
@char0n, would you mind opening an issue for this in fantasyland/fantasy-laws?

@davidchambers copypasted the issue to fantasyland/fantasy-laws. Do you want me to provide a PR there ?

Do you want me to provide a PR there ?

Don't worry. I've added an item to the to-do list in fantasyland/fantasy-laws#1. I'll close this issue since we're now tracking it elsewhere. :)


OpenGL full screen not centered in MacOS

When pressing F11 to switch to full screen, the screen is not centered in OpenGL in MacOS (displayed on the left side instead of centered). It is centered in SDL and GTK+.

This happens also on Windows. It started occurring with my first fork release.
Last official GDash correctly centered OpenGL, but there were a lot of code changes by Zoltan and many 3rd party library updates.
I'd love to fix this, even more it would improve the "full cave view" I'm still working on.
I guess it has something to do with glViewport() and glOrtho().

If I remove glViewport() completely the game is centered in fullscreen, but also looks like it was stretched.

Stretched is not good, aspect ratio should be correct.

Can you tell my why full screen size on Windows/Linux stretches the graphics to fit the screen, while on Mac it is not stretched, but just adds black areas around the graphics?
So my approach would be:

Add new setting "Center OpenGL full screen"
Ignore user scaling factor, use calculated one. Don't call glViewport() to get centered.
The game gets scaled preserving the aspect ratio.
Find out why intermissions are scaled differently and not preserving the aspect ratio. Have you noticed the missing pixel line at the top and the bottom?

That’s one of the mysteries that would really be interesting to know😎
Black bars should be only left and right, or did I misunderstand something?

I just read the manual of glViewport. As far as I understand this, the initial values are 0,0 for the lower left corner - which would explain why it’s not centered.

I just read the manual of glViewport. As far as I understand this, the initial values are 0,0 for the lower left corner - which would explain why it’s not centered.

It's just the start coordinate. The full screen flag should just do what it says... But why the hell does it work on Linux and not Windows/Mac?

New release is out. Let me know what you think. Note that the user defined scaling factor is relevant.

Generally a good workaround! But I’m not fully happy with the slightly stretched screen as there are more visual distortions when using shaders like crt-geom so a correct aspect ratio would be nicer (with black bars left and right on a 16:9 display). But at least this option can be switched on/off…😎

There are a small repeatable bug:

Go to Options and set

Scaling factor: 5
Full cave view: yes
OpenGL: Center full screen: no

Play a full screen game. Visual area aligns left as expected
Go to Options and set:

Full cave view: no

Play a game. Everything looks as usual
Go to Options and set:

Full cave view: yes

A new game starts on centered screen
Switching "OpenGL: Center full screen: no/yes" doesn't affect the result: every new full screen game starts on centered screen and only cold start of GDash helps.

Not a big deal, just a minor issue.

To be honest, I would rather always have a cantered full screen and an option to stretch a visual area to bounds (which may causes visual distortions) or do not stratch and have a precise visual area aspect ratio (with black bars left and right).
But if this is difficult to implement for some reason, the current version's solution is also a pretty good workaround.

@renyxadarox wrote

There are a small repeatable bug
[...]
every new full screen game starts on centered screen and only cold start of GDash helps.

I could reproduce it on Windows. IMHO this is an OpenGL bug or feature. I guess that it's not centered at the beginning is the bug. My setting just adds a workaround...

I guess that it's not centered at the beginning is the bug. My setting just adds a workaround...
Does it make sense to have the option "OpenGL: Center full screen" at all?
In my opinion, few people need to have full screen left alignment.
Most people would be happy with always centered full screen.

Actually, no matter what is a value of "OpenGL: Center full screen" option, the full screen view is always centered now and it affects only visual area aspect ratio:
OpenGL: Center full screen: no
1680x1050

OpenGL: Center full screen: yes
1680x1050

So, maybe you should rename it as "Full screen: Stretch to bounds"? Because that's exactly what it's doing right now.
In Windows at least. Maybe this option works differently in McOS, I don't know.

Maybe Czirkos Zoltan can help?
Or maybe there are websites where these things could be asked? I’m no programmer unfortunately…

I have updated SDL: mingw-w64-x86_64-SDL2 2.26.5-1 -> 2.28.2-1
No change. That's all I can do for now.

To preserve the aspect ratio you need to set a higher scaling factor. Forgot to mention that.

Now I know that this happens on all platforms and if it happens depends on the screen size.
I'm going to calculate the perfect scaling factor.  SDL will add only one small black bar somewhere, but this preserves the aspect ratio.
Enabling the setting opengl_center will stretch the image almost unnoticeable. So maybe we can remove the setting completely and preserve the aspect ratio.

I though I have fixed it, now my Linux laptop does cut off the score for 40x22 view. (Release 1.9.5)
I have checked when this happens with gdash -v: (Tried different view sizes)
Values for h_padding that work correctly: 70, 132
Values for h_padding that don't work correctly: 245, 210, 160
So the conclusion is: Whenever h_padding is at least 160 it is a problem??


namespace unique error

Coverage increased (+0.06%) to 94.475% when pulling d7389c9b0f1ff073f9edc5b53e13579474845304 on cycgit:master into 718d0eb7a9a7e95f67fb5cc126f7dc653945a817 on dvajs:master.

https://github.com/dvajs/dva/blob/master/src/createDva.js#L77-L84，把这段删了吧。重复的判断确实放在 checkModel 会比较好。

Coverage decreased (-0.1%) to 94.318% when pulling e8f2bb59953e4bf0b0312ffb5f5b5ed73dda3733 on cycgit:master into 718d0eb7a9a7e95f67fb5cc126f7dc653945a817 on dvajs:master.

Coverage decreased (-0.1%) to 94.318% when pulling e8f2bb59953e4bf0b0312ffb5f5b5ed73dda3733 on cycgit:master into 718d0eb7a9a7e95f67fb5cc126f7dc653945a817 on dvajs:master.

💯


Adding new driver for SG384

Beta version driver for stanford SG384.
@giulioungaretti @alexcjohnson
Implemented most commands from the manual.
Any feedback?

@WilliamHPNielsen changed the units part. I think we can merge. :dancer: ?


Request never finish if error is thrown in async function

Here is a minimal case
@app.Route ('/testErro1')
@Encode()
testError1 () async
{
    var resp = await testError2();

    return resp;
}

@app.Route ('/testErro2')
@Encode()
testError2 () async
{
    throw new app.ErrorResponse(400, {"error" : "invalid id"});

    return new BoolResp()
        ..value = true;
}

I created this plugin to handle this
class Catch {

  const Catch();

}

void ErrorCatchPlugin(app.Manager manager) {

    manager.addRouteWrapper(Catch, (metadata, Map<String,String> pathSegments,
                                    injector, app.Request request,
                                    app.RouteHandler route) async {

        try
        {
            var result = route(pathSegments, injector, request);

            if (result is Future)
                return await result;
            else
                return result;
        }
        catch (e, s)
        {
            return new Resp()
                ..error = "$e $s";
        }

  }, includeGroups: true);

I love plugins! However, Redstone should handle this by default.

Thanks for reporting.

I've released v0.5.21 with a fix for this. Please confirm.

Worked :)


Fix issue #398 Remove Github Integration folder

Fix issue #398
remove github integration folder.

:+1:


Alohomora doesn't close doors

Hi
After updating to 1.13 and updating to the latest version of magic, I found out Alohomora doesn't close the doors it opens. This is my configuration for alohomora:
alohomora:   parameters:     bypass_build: true     destructible: doors   worth: 10
Is there something I can do about it?
Thanks!

I don't think there's anything you can do about this... the undo system will need to be updated to store extended block state, since doors don't use their data bytes anymore.
I will leave this open in hope of fixing it at some point.

Thanks! But this currently breaks the gameplay on my server. I'm using this code right now to easily close and open doors, if you'd like to have it.
`	public static void setDoorOpen(Block block, boolean open) {
BlockData data = block.getBlockData();
if(block.getBlockData() instanceof Openable) {
if(!((Openable) data).isOpen() && open) {
((Openable) data).setOpen(true);
			block.setBlockData(data);
		} else if(((Openable) data).isOpen() && !open) {
			((Openable) data).setOpen(false);

			block.setBlockData(data);
		}
	}
}`

I think I've got this fixed for you. Unfortunately it didn't quite make it into 7.7.5 but you can grab the dev build here:
http://jenkins.elmakers.com/job/MagicPlugin/2896/com.elmakers.mine.bukkit.plugins$Magic/

It's fixed indeed! Thank you very much for helping me this quickly!


Split munki recipes into download + munki recipes.

For new identifiers, used Sam's standard com.github.keeleysam.autopkg.download.NAME format.
Fixes #49.

@keeleysam Good to go!


Need to check hashes of downloads before installing them in ~/.kuberlr/.../

I switched to using version 1.19.10 from 1.21.x of kubernetes and the linked kubectl stopped working. When I looked at the storage directory I saw this:
$ ls -l ~/.kuberlr/darwin-amd64
total 368352
-rwxr-xr-x  1 ericp  staff  49577376 14 Jun 17:49 kubectl1.17.3
-rwxr-xr-x  1 ericp  staff  28015453 17 Jun 11:45 kubectl1.19.10
-rwxr-xr-x  1 ericp  staff  55831200 14 Jun 17:50 kubectl1.21.1
-rwxr-xr-x  1 ericp  staff  52540272 16 Jun 14:32 kubectl1.21.2

I redownloaded version 1.19.10 and got a file with this size:
rwxr-xr-x  1 ericp  staff  49397776 14 Apr 22:07 kubectl

And verified it:
echo "$( curl -sL 'https://dl.k8s.io/v1.19.10/bin/darwin/amd64/kubectl.sha256') kubectl" | sha256sum -c

Working on a PR now

Fix being worked inside of this PR https://github.com/flavio/kuberlr/pull/10

Fixed starting from https://github.com/flavio/kuberlr/releases/tag/v0.3.2


A collection of small stories

See individual commits for details

 Make "Background Information" a required field

 Submit a Paper Petition should match online submission text fields

 Hide "Special Consideration" in Back End

 Hide "Committee Notes" in Back End

 Backend "Update ABMS link" to read "ScotParl Weblink"


What makes NodeOS better than other traditional operating systems?

General purpose operating systems are extremely complex. First they have to provide an extensive HAL (Hardware Abstraction Layer) that can support the wide range of different devices they can be installed on. Second, they need to provide functionality as well as protection for multiple users and many programs/services that support those users.
Web servers require none of that functionality. They generally run off of VMs (Virtual Machines) that abstract away all of the hardware-specific details and they're generally designed for single-purpose usage.
Lets say you decided to build an OS from scratch. Start by stripping everything down to the bare essentials. HAL isn't required because the VM already takes care of abstracting away the hardware. V8 already provides memory/filesystem sandboxing so the separation of user/kernel space (and all of the processing overhead it requires) is no longer necessary.
What you get is an OS that takes up less than 10MB of space, boots up in seconds, requires zero maintenance (other than pushing new updates), requires minimal memory overhead, and runs as fast or faster than the fastest C implementations running on a general purpose OS.
If you've ever used IaaS, you've probably had to deal with the pains of deployment. Deploying a complete general-purpose OS image is a PITA. Docker makes things easier by making it so you only have to deploy your application-specific differences but it adds yet another layer of abstraction, including more complexity and space/memory overhead. Plus, you still have to manage updating the underlying OS for security vulnerabilities and bug fixes.
Here are the layers of abstraction for a Docker deployment:
Hardware -> VM -> Kernel Space -> User Space ->  Docker -> V8 VM -> Application

And for an application specific system in NodeOS:
Hardware -> VM -> Kernel Space -> V8 VM -> Application

Fewer layers, means less bloat and drastically reduced surface area for security vulnerabilities.

Use case - Building a Load Balancer:

clone NodeOS
install a load balancer implementation via NPM
install additional dependencies to auto-restart, log, profile the system
add a custom configuration

Run the build step which packages everything (incl a minimal Linux kernel and Node/V8) as a VM. Then deploy the VM using git or some other deployment tool.
Occasionally, you may need to update dependencies (npm update) and/or modify the config. Run the build step again and it's ready for deployment.
In most setups, the OS lives on the server and changes/updates are either managed remotely via provisioning tools (ex ansible/puppet) or pushed as Docker containers.
In a Unikernel. The setup, dependency management, build process can all be automated using NPM and existing libraries. The builds are small enough that they can be versioned using git.
Theoretically, you could do a whole system setup with 4 commands:

git clone
npm run install
npm run build
npm run deploy

JavaScript.

The thought of having a NodeJS REPL as my command line / operating system reminds me (very fondly) of playing with my Apple //e, with BASIC right at the command line.

omg you're right!
How do you do a directory listing from the REPL?
---- On Wed, 06 Jan 2016 04:43:55 +1000 Hans Jorgensen<EMAIL_ADDRESS>----
The thought of having a NodeJS REPL as my command line / operating system reminds me (very fondly) of playing with my Apple //e, with BASIC right at the command line.
—
Reply to this email directly or view it on GitHub.

@jwulf in nsh (the command prompt in NodeOS) the commands are the same as a standard Linux terminal. "ls" lists the folders and files in a directory. If you're talking about the default JS REPL, it evaluates JS so you'd actually have to pop a node script in there to read and print the directly files and folders.


Add placeholder prop and placeholder.

Adds a placeholder prop for adding a placeholder to the input.

Already available via inputProps.


Visualization omitted, Javascript library not loaded! in Google Colab

Code used,
background = X_train
explainer = shap.DeepExplainer(yx_model, background)

data_for_prediction = X_test[1:2]
print( "Data for prediction: {}".format(data_for_prediction) )

shap_values = explainer.shap_values(data_for_prediction)
shap.force_plot(explainer.expected_value[0], shap_values[0], data_for_prediction)

Get this result,
Visualization omitted, Javascript library not loaded!
 If this notebook was from another user you must also trust this notebook (File -> Trust notebook). If you are viewing this notebook on github the Javascript has been stripped for security. If you are using JupyterLab this error is because a JupyterLab extension has not yet been written.

How to solve?

Looks like Google Colab needs shap.initjs() in every cell where there is a visualization.

Thanks for the heads up!

Does it work with using shap.initjs() in every cell in Google Colab?

Does it work with using shap.initjs() in every cell in Google Colab?
Yes, it does work

Error in Colab : Visualization omitted, Javascript library not loaded!  If this notebook was from another user you must also trust this notebook (File -> Trust notebook). If you are viewing this notebook on github the Javascript has been stripped for security. If you are using JupyterLab this error is because a JupyterLab extension has not yet been written.

@slundberg @shivam1702 I got error 'Visualization omitted, Javascript library not loaded!' then I run shap.initjs() but still it did not work. Is there any code to run for databricks notebook to visualise the force_plot?

Yes it runs when you use shap.initjs() in the same cell as the visualization function it will run.
Like this for example:
shap.initjs(),
shap.force_plot(base_value=explainer_shap.expected_value,
shap_values=shap_values[10,:],
features=X_train.iloc[10,:])

@slundberg @shivam1702 I got error 'Visualization omitted, Javascript library not loaded!' then I run shap.initjs() but still it did not work. Is there any code to run for databricks notebook to visualise the force_plot?

Facing the same problem with databricks notebook. Running shap.initjs() in the same/every cell doesn't solve the problem.

I got a solution for Databricks. You just need to get the HTML out of force_plot and use displayHTML to see the force_plot.
Try this:
force_plot = shap.force_plot(xgb_explainer.expected_value, xgb_shap_values, X_test, matplotlib=False)
shap_html = f"{shap.getjs()}{force_plot.html()}"
displayHTML(shap_html)
more info: https://towardsdatascience.com/tutorial-on-displaying-shap-force-plots-in-python-html-4883aeb0ee7c

You need to import matplotlib. Example:
import matplotlib
shap.force_plot(explainer.expected_value, shap_values[0, :], X_display.iloc[0, :], matplotlib=matplotlib)

@abai05 it worked for me. Thank you so much!

@abai05  legend! thanks


Infinite scroll compenent is added

Issue reference - #185
Title
I have added infinite Scroll compenent
Proposed Changes

Check List (Check all the applicable boxes)

[x] Bug(s) fixed
[ ] Enhancement done
[ ] New Feature(s) added
[ ] Documentation changed
[x] All new and existing tests passed.
[x] My code follows the code style of this project.

Screenshots (if any)

Additional Information

@thekavikumar please review my pr I had done some changes

@thekavikumar please review


AES - compilation warnings

Hi
In VCS I get the following 2 warnings:

Warning-[SIOB] Select index out of bounds
../../../../../../opentitan/hw/ip/aes/rtl/aes_key_expand.sv, 218
"out_prd[(0 - 1)]"
The select index is out of declared bounds : [3:0].
In module instance : u_aes_key_expand
In module : aes_key_expand.
Warning-[SIOB] Select index out of bounds
../../../../../../opentitan/hw/ip/aes/rtl/aes_sub_bytes.sv, 71
"out_prd[0][(0 - 1)]"
The select index is out of declared bounds : [3:0].
In module instance : u_aes_sub_bytes
In module : aes_sub_bytes.

Are you aware of these ?
thanks

Hi @arnonsha ,
thanks for reporting this issue. I don't understand why VCS sees a problem here. The RTL code looks as follows (second warning):
  for (genvar j = 0; j < 4; j++) begin : gen_sbox_j
    for (genvar i = 0; i < 4; i++) begin : gen_sbox_i

      // Rotate the randomness produced by the S-Boxes over the columns but not across rows as
      // MixColumns will operate across rows. The LSBs are taken from the masking PRNG (prd_i)
      // whereas the MSBs are produced by the other S-Box instances.
      assign in_prd[i][j] = (j == 0) ? {out_prd[i][3],   prd_i[i][j]}  :
                                       {out_prd[i][j-1], prd_i[i][j]};

So, whenever j is zero, we use out_prd[i][3] instead of out_prd[i][0-1] (what VCS thinks). Also I don't see this warning locally (VCS R-2020.12-SP2).

This is strange
I am running the same version and get this warning, without any special flags.
According to documentation there's a "-boundscheck" option that can be used. Can you please try at your end ?

Oh, checking the logs again I can see the warnings, too. After discussing with @rswarbrick it looks like VCS has a problem with this ternary. I will prepare a PR to fix it.


"namespace "std" has no member "shared_mutex" " despite including shared_mutex

Issue Type: Bug
using gcc 9.3.0, the following code states namespace std has no member shared_mutex:
#include <shared_mutex>

using read_lock = std::shared_lock<std::shared_mutex>;
using write_lock = std::unique_lock<std::shared_mutex>;

shared_mutex is included, so this behavior is not expected. Intellisense fails on my project teammate's computer as well.
It's possible that something is incorrectly set in my VS Code settings, but I've done two projects in the past two months each about 1000-2000 lines each that have used C++17, and I haven't gotten errors about it. So I wonder if something strange is going on specifically with shared_mutex.


Is it possible to have an https service check

I tried and this didnt work
SERVICE_3000_CHECK_HTTPS=/api/status

but if i use
SERVICE_3000_CHECK_HTTP=/api/status

it fails because it is only listening on an https server

I believe this is resolved by #388


[BEAM-214] Add ParquetIO

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[X] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[X] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[X] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[X] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

This is a first simple version of the ParquetIO dealing with Avro GenericRecord.
I have the following TODO list on this one:

the Write (test) has to be fixed with the direct runner.
improve (maybe providing a custom ParquetReader) the Read to use a BlockBasedSource to be able to split the read on different sources (using offset, ...).

Coverage decreased (-0.003%) to 69.751% when pulling 28c897b4827fa5dfc394fec6a7e5176e0ad6266b on jbonofre:BEAM-214-PARQUETIO into 1c6e667414788fe99f583fac39d458a4984ae162 on apache:master.

Did you want a reviewer for this, or is this just to exhibit the code? If you did want a reviewer, R: me?

Hey Dan. The PR is not fully ready, but you can already take a look in your have some spare time ;) Thanks !

Hi guys, just checking, any updates perhaps?

Resuming my work on all PRs

Hey dude, I'm already using this code for production, however I added some more to allow parameterizing.

Using specific datum encoding is necessary to support 1.8.1's logical type conversions.
Parameterizing helps with that but also by not having to cast SpecificRecords in another SerDe step in your beam since the ParquetReader/Writer already handles that.

I also think you should at least document how to pass hadoop configuration properties since that's used by parquet and doesn't go through the usual beam options with these code.
I can help contribute some code if you want.

Hey !
Agree with your comments/changes ! Let me improve the PR !
You can also submit PRs on my branch !
I'm resuming my work on this PR (and also others ;)).
Thanks !

@Igosuki could you please share your improved version?
Thank you!

Any updates? IMHO, without parquet, batch mode (and/or spark runner) is mostly impractical. However, with parquet, Cloud Storage (as data lake) and DataFlow( with the new Apache Beam SQL DSL) could be a great combo.

I plan to resume my work on this PR later this week.

Also very interested in this PR, look forward to more updates!

I'm on the top of my TODO for the following days (I just updated AmqpIO). Stay tuned !

Any updates perhaps?

Yup. Starting the rebasing and update.

@jbonofre, thanks for the update!

I did some analysis on what goes wrong writing to a Parquet file.
First I added the option present in the Parquet write to overwrite the output.
That yielded a new insight into what goes wrong.
It turns out that the current implementation uses the same filename for each parallel instance writing the data. That is why the test fails with file already exists.
With the overwrite option you therfor get a different error that effectively says "bad file" which is simply caused by the threads obliterating the data written by the others.
My current assessment is that the "numShards and "shardNameTemplate" construct (as seen in TextIO) needs to be added here as well to make it work.

I did a local attempt to use FileBasedSink to make this work because that already has the notion of numShards.
As far as I can tell this won't work for Parquet because I think that this assumes a header, set of records and a footer. The desired number of shards is later on realized by rewriting to the desired number of files after everything has been completed.
If I am right here then formats like Parquet and Orc (i.e. blocks of mangled records with advanced stats in headers and footers) will be very hard to created in a controlled way in Beam.

Rebased and updated to Beam 2.2.0-SNAPSHOT.

@nielsbasjes I tried to use block and file based sink/source first in AvroIO. Unfortunately, the Parquet API and Beam don't match so well ;), so I stayed at a more "higher" level.

retest this please

Coverage increased (+0.01%) to 69.691% when pulling 355b0a73788b3ba378992814bab7a020d6b4da13 on jbonofre:BEAM-214-PARQUETIO into 8b540d2dd43af2a0495884d8152472a7ebff8a8e on apache:master.

R: @jkff
Eugene, can we chat a bit about this PR ? I would like to move forward on this one and I would love to have your input and comments. Thanks !

Sure, what would you like to discuss? I'd like to understand more about the use cases of Parquet in the wild to decide whether the proposed API in this PR is the right API - is there any Parquet expert you can add also as a reviewer? Eg I suppose people might want to choose to read only a subset of columns, or use filter pushdown. Write might need to be configurable in some way too.
Other notes to modernize this PR:

The read transform seems to be non-parallel and reads a single file. Is this intentional?
I would recommend to start right away with a readFiles() version, like in TextIO, so that this can be freely combined with FileIO.match() and FileIO.read(). Is it possible to read chunks of a single Parquet file in parallel? If yes, then might want to use ReadAllFiles and have a BlockBasedSource, but might also just start with a ReadFn (but on top of ReadableFile).
Likewise I would recommend to implement the write side just as a sink on top of s.apache.org/fileio-write when it is ready (which should be soon enough so I think it makes sense to wait).

Parquet is used to serialize and compress proto, thrift and avro files into
queryable column oriented files.  The hard part is that it uses a very api
private internal reader to do a break down and parallelize reading. So the
code to split the input source like with other beam io would have to be
custom
On Thu, Sep 7, 2017, 8:43 PM Eugene Kirpichov<EMAIL_ADDRESS>wrote:

Sure, what would you like to discuss? I'd like to understand more about
the use cases of Parquet in the wild to decide whether the proposed API in
this PR is the right API - is there any Parquet expert you can add also as
a reviewer? Eg I suppose people might want to choose to read only a subset
of columns, or use filter pushdown. Write might need to be configurable in
some way too.
Other notes to modernize this PR:

The read transform seems to be non-parallel and reads a single file.
Is this intentional?
I would recommend to start right away with a readFiles() version,
like in TextIO, so that this can be freely combined with FileIO.match() and
FileIO.read(). Is it possible to read chunks of a single Parquet file in
parallel? If yes, then might want to use ReadAllFiles and have a
BlockBasedSource, but might also just start with a ReadFn (but on top of
ReadableFile).
Likewise I would recommend to implement the write side just as a
sink on top of s.apache.org/fileio-write when it is ready (which
should be soon enough so I think it makes sense to wait).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/beam/pull/1851#issuecomment-327888785, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAYyhtyUeewbaogiwCDHlCfawOCIv7REks5sgDlegaJpZM4LupIS
.

A complete parquet implementation would allow pushing down the FilterApi and splitting the workload using the same kind of method that's used in io's like hadoop-file-system not the internal reader as seen here https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java#L141.
I think Beam should read the footers itself to determine how to parallelize the workload.

Coverage increased (+0.003%) to 69.509% when pulling 5fdc9b284a35cf3d342deb2430fd2cc73b5018ca on jbonofre:BEAM-214-PARQUETIO into 50532f0a92d7ce8dbbdc6c3179ab7b9efde6a746 on apache:master.

I've started to implement a ReadFiles wrapping ReadAllViaFileBasedSource. Once done, I will PoC to implement sharding based on the footer.

Is this PR in active development? Would it make sense to close it for now and reopen when there's something ready to be reviewed?

I'm working on readFiles important now. This code is already used by some users. If you don't mind, I prefer to keep this PR open as I think I'm pretty close of the readFiles impl.

Rebased and added the gradle build. I'm working on adding the readAll pattern support.

Alright, let me know when it's ready for review.
For now I have a few comments:

The interface seems very similar to AvroIO. That means perhaps it should be pretty much identical (e.g. allow reading using Avro generated classes, allow not specifying the schema etc.) - WDYT? Or perhaps replicating the API service of AvroIO would be too much work - but then what would be the minimum useful subset?

Splitting within a file appears to be lacking, which could be a serious performance issue. It would be good to have at least basic splitting into reasonably-sized chunks, even if not dynamic splitting (which would require BoundedSource and going via ReadViaFileBasedSource).

I will resume my work on it today after RC2 on vote. Thanks !

@eugene Parquet can be deserialized in several formats including Avro, I
don't think it's a good idea to tie it to Avro, instead users should be
able to use the Parquet read support implementations to map to values of
their choosing be it Avro or Protobuf messages.
On Tue, Feb 13, 2018, 9:05 PM Jean-Baptiste Onofré<EMAIL_ADDRESS>wrote:

@jbonofre commented on this pull request.
In
sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java
https://github.com/apache/beam/pull/1851#discussion_r167985298:

     GenericRecord read;

     while ((read = reader.read()) != null) {

       processContext.output(read);

     }

   }

 }

}

}

/**

{@link PTransform} writing a {@link PCollection} of {@link GenericRecord} into a Parquet file.

*/
@AutoValue
public abstract static class Write extends PTransform<PCollection, PDone> {

Agree. I'm figuring out a new to use the channel with the ParquetWriter
to be use in open() method.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/beam/pull/1851#discussion_r167985298, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAYyhqUL3-eQf4dhKOW5M0VDcvsUQyJoks5tUeskgaJpZM4LupIS
.

Although it's likely most of the users today would use Generic Records as
the main Serde format of Parquet
On Tue, Feb 13, 2018, 10:44 PM Guillaume Balaine<EMAIL_ADDRESS>wrote:

@eugene Parquet can be deserialized in several formats including Avro, I
don't think it's a good idea to tie it to Avro, instead users should be
able to use the Parquet read support implementations to map to values of
their choosing be it Avro or Protobuf messages.
On Tue, Feb 13, 2018, 9:05 PM Jean-Baptiste Onofré <
<EMAIL_ADDRESS>wrote:

@jbonofre commented on this pull request.
In
sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java
https://github.com/apache/beam/pull/1851#discussion_r167985298:

     GenericRecord read;

     while ((read = reader.read()) != null) {

       processContext.output(read);

     }

   }

 }

}

}

/**

{@link PTransform} writing a {@link PCollection} of {@link GenericRecord} into a Parquet file.

*/
@AutoValue
public abstract static class Write extends PTransform<PCollection, PDone> {

Agree. I'm figuring out a new to use the channel with the ParquetWriter
to be use in open() method.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/beam/pull/1851#discussion_r167985298, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAYyhqUL3-eQf4dhKOW5M0VDcvsUQyJoks5tUeskgaJpZM4LupIS
.

@igosuki Thanks, that's a reasonable point, do you have a suggestion on how to incorporate this into the current API? Is it applicable on the write side too, I suppose?
I looked at the ReadSupport and WriteSupport classes. They seem quite tied to the Hadoop API and I'd prefer to not expose anything related to Hadoop from the Beam API surface - maybe we should just implement the main case of readAvro() for now, hiding the Hadoop details under the hood, and meanwhile think how to expose it more generally?

@jkff Likely something like :
myParquetIO.withReadSupport(ReadSupport<T>)

Where T has to implicitly respect the same extend clause as the one of the extending ReadSupport class so something like :
ParquetIo
.read("hdfs://myparquet")
.withReadSupport(ProtobufReadSupport.class, MyMessage.class)
...
.withProtobufCoder(ProtobufCoder.of(MyMessage))

Then it would yield here https://github.com/apache/beam/pull/1851/files#diff-03b8632d3bc8da69525461ba588f949bR330
static class ReadFn extends DoFn<FileIO.ReadableFile, GenericRecord> {
   @ProcessElement
   public void processElement(ProcessContext processContext) throws Exception {
     Path path = new Path(processContext.element().getMetadata().resourceId().toString());
     try (ParquetReader<T> reader = new ParquetReader(path, readSupport);
       T read;
       while ((read = reader.read()) != null) {
         processContext.output(read);
       }
     }
   }
 }

Agree, but it doesn't solve the reading/writing part. For this part, I'm working on using low level surface Parquet API.

I was looking at this PR as I hope to use this in the near future :) Would it be possible to include full ValueProvider support so this can be templatized easily? I noticed that some of the fields (schema?) in the Builder does not provide a ValueProvider option.

Yes, but the blocking point now is not about the format/value provider, it's the core reader/writer based on FileIO. I sent an e-mail on Parquet mailing list, and I'm evaluating a workaround implementing our own Parquet reader/writer.

It would be awesome if the parquet reader could split work according to parquet file row group indexes. Is this what you have in mind @jbonofre ?

@Igosuki exactly, that's the purpose !

What's the progress here? I'm willing to help if necessary. Cheers

@Fokko did you see my message on the dev mailing list about Parquet ? Basically, I asked Parquet guys if it would be possible to have a new release to smoothly use Beam FileIO.

For all watchers on this PR, we have basically three approaches:

If you take a look on the first commits in the PR, ParquetIO didn't leverage FileIO (because it didn't exist in the early stage). It was a complete IO with the "split" was limited (even if it worked fine for both read and write).
To be able to leverage FileIO, we need a way to use NIO Channel for Read. That's possible (via InputStream) with Parquet master as they expose the API (take a look on the mail I sent on the dev mailing list for details). This approach requires a new Parquet release.
Write our own Parquet reader/writer (low level, a bit like we did for Avro). It requires more work and I'm not so comfortable with this approach especially to leverage features and fixes provided by Parquet.

So, basically, I would love to hear your thoughts about ParquetIO: would you like a first simple ParquetIO in Beam even if the split is not optimal or if it's not based on FileIO, or are you OK for wait to have a FileBased one ?
By the way, I'm resuming my work on this one now that I'm back from travels.

I already tested this code in production, and another approach using Hadoop
input file IO and it works but not as well as delimiter based file IOs.
We should be able to create splits based on row groups.
I think what you need is something leveraging the latest features from S3A
(fadvise, head skip) and splits with hints for row groups, and obviously
including predicates and projections.
Right now for me, it's not satisfactory because with large parquet files
batch processes quickly become bottlenecked and parallelism is very
limited. It's frustrating that Spark is still better at this.
On Mon, Mar 19, 2018, 4:23 PM Jean-Baptiste Onofré<EMAIL_ADDRESS>wrote:

For all watchers on this PR, we have basically three approaches:

If you take a look on the first commits in the PR, ParquetIO didn't
leverage FileIO (because it didn't exist in the early stage). It was a
complete IO with the "split" was limited (even if it worked fine for both
read and write).
To be able to leverage FileIO, we need a way to use NIO Channel for
Read. That's possible (via InputStream) with Parquet master as they
expose the API (take a look on the mail I sent on the dev mailing list for
details). This approach requires a new Parquet release.
Write our own Parquet reader/writer (low level, a bit like we did
for Avro). It requires more work and I'm not so comfortable with this
approach especially to leverage features and fixes provided by Parquet.

So, basically, I would love to hear your thoughts about ParquetIO: would
you like a first simple ParquetIO in Beam even if the split is not
optimal or if it's not based on FileIO, or are you OK for wait to have a
FileBased one ?
By the way, I'm resuming my work on this one now that I'm back from
travels.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/beam/pull/1851#issuecomment-374251378, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAYyhiUODG1Ql0EdRAJa9HkBxIK3nNyPks5tf82AgaJpZM4LupIS
.

As a first approach, we can mimic what Spark does here.

@lgajowy by the way, does it mean that you already used the first "stage" of this PR ? I'm curious ;)

@jbonofre I did when you started it, once to create a Druid segment
rewriter and another to test performance of a simple windowed sum over a
large number of keys vs Spark.
My files go from small files (due to people having implemented constant
partitionning with different sized keys) to 400Mo parquet files.
I tested your code vs avro io vs the recent Hadoop input split over a few
TBs.
In the end, I had to fix coders which catch EOF because Flink throws EOF to
trigger spill on disk and reading parquet quickly overflows the heap...
The code works fine, but Hadoop input format io over S3A pefrozm better. I
think we need some examples to attain the same performance as Spark group
by + partition when there is a large cardinality over keys reading binary
formats.
On Mon, Mar 19, 2018, 6:20 PM Jean-Baptiste Onofré<EMAIL_ADDRESS>wrote:

@lgajowy https://github.com/lgajowy by the way, does it mean that you
already used the first "stage" of this PR ? I'm curious ;)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/beam/pull/1851#issuecomment-374294586, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAYyhh-hh3qzHIlJ0PNCZcFgTHr02wRPks5tf-jZgaJpZM4LupIS
.

@lgajowy thanks a lot for the details !!!! Do you mind to quickly describe (with a pipeline example) your "perfect target" ParquetIO ? I can implement that way as a first step. Thoughts ?

Reviewed 2 of 5 files at r1, 2 of 3 files at r2, 3 of 3 files at r3.
Review status: all files reviewed at latest revision, all discussions resolved, some commit checks failed.

Comments from Reviewable

I'm closing this PR as it has been included in the one from @lgajowy


Explore making it clearer that password can also be used instead of a PIN

Currently, during the setup process, you're asked to setup a PIN, and there is a "screen lock options" text that once tapped allows you to switch between PIN and password. This may not be immediately clear to everyone.
Possible options to improve UX:

Change the string "screen lock options" to "use a password instead" and "use a PIN instead" when on the password screen to make it more obvious.
Introduce an additional screen before PIN input screen that clearly lays out PIN and Password as the available options and lets the users choose one of the two. If we go for this option, we could include some information about each such as that we recommend at least 6 digits for a PIN and 18 character passwords or 7 diceware word passphrases. We can also explain what the maximum length of a password is (64 characters).

I lean towards option number 2 as it makes the options clear along with providing more information to help the user choose according to their needs.

the upstream PIN/password screens already include some information, although in brief.
option 2 above suggests more detailed information. we could go with option 2 if such detailing is needed, or else option 1 is more optimal as it provides minimal needed instructions and reduces a step.
currently upstream PIN entry screen already recommends 6 digits.
password entry doesn't recommend 18 digits, we can just change that message.


Support of Aggregator in Camel extensions

Aggregator is not yet implemented for Camel generation & parsing.
Thus because in Camel, Aggregator cannot be directly mapped to a specific endpoint but should be considered as a property of other endpoints in the route definition (ex: split or multicast). This led to a difficult implementation ... but have no fear!

Done!


Add Genes For Good reader

Hi there!
I found this codebase to be very useful for the project for us at Sano Genetics. Thank you for open sourcing it!
I have added an extra reader for genotype files from "Genes For Good" along with a test case for it.
I hope it is clear, and in line with the existing code style. Let me know if there are any changes you'd like me to make. I'm also happy to contribute to this repository in the future if you need some help with it.
Will

Thanks @willgdjones. Glad to hear that the project is useful, and thanks for the contribution! Any help is welcome and appreciated.
This PR looks great, and I'll get it merged to master soon!


[Service Bus] Fix xml comments to properly resolve the reference

I stumbled over a bunch of CS warnings related to documentation and figured I fixed them quickly.
Namely

CS1580 Incorrect xml signature
CS1574,1584,1581,1580 Cannot resolve reference in XML comment

Can one of the admins verify this patch?


[Bug]

Description
Steps to Reproduce

Expected Behavior
Actual Behavior
Basic Information

Version with issue:
Last known good version:
IDE:
Platform Target Frameworks:

iOS:
Android:
UWP:

Android Support Library Version:
Nuget Packages:
Affected Devices:

Workaround

Reproduction imagery

Reproduction Link

Not sure what the intention was, but this doesn't look right :)

Not sure what the intention was, but this doesn't look right :)

Ooops! Sorry all!

Ooops! Sorry all!


Deno support in dev

You can run the app with deno after it is built but deno task dev does not work. This seems to be the issue, have to revisit after it is closed.
https://github.com/denoland/deno/issues/23326

Fixed by #16


Merge change into already existing output when possible

The proposed changes do the following:

If the change address is already included in one of the outputs, the change (minus the fee) will be added to last output directed towards the change address.
If no outputs are provided but a change address is provided, the entirety of all input UTxOs will be sent to the change address
If you want to force the change address into a separate output, create a dummy output like so as the last output added builder.add_output(TransactionOutput(change_address, 0)). This will force the change into a new output UTxO even if the change address is already included among the outputs.

FYI I also added a method to transaction.Output called copy() which essentially deep copies itself. I found it useful for estimating the fees but I'm definitely open to other ways of doing this.
@cffls I updated all the relevant tests in test_txbuilder.py to reflect the new fees and outputs. If you have the chance to take a look and make sure I didn't fundamentally alter the purpose of any of them...
Especially test_tx_add_change_split_nfts as I couldn't figure out exactly what it was trying to achieve.

Codecov Report

Merging #38 (7f12a8c) into main (f5e03cc) will decrease coverage by 0.48%.
The diff coverage is 40.00%.

@@            Coverage Diff             @@
##             main      #38      +/-   ##
==========================================
- Coverage   85.82%   85.33%   -0.49%
==========================================
  Files          20       20
  Lines        1940     1930      -10
  Branches      407      408       +1
==========================================
- Hits         1665     1647      -18
- Misses        208      212       +4
- Partials       67       71       +4

Impacted Files
Coverage Δ

pycardano/txbuilder.py
90.36% <33.33%> (-1.83%)
:arrow_down:

pycardano/transaction.py
90.86% <66.66%> (-0.51%)
:arrow_down:

pycardano/address.py
70.78% <0.00%> (-1.56%)
:arrow_down:

pycardano/serialization.py
87.44% <0.00%> (-0.23%)
:arrow_down:

pycardano/plutus.py
92.99% <0.00%> (-0.14%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f5e03cc...7f12a8c. Read the comment docs.

Thanks for the really insightful comments, @cffls. I'll go ahead and get to work on this and let you know if I have any followup questions.

Btw, integration tests failed. It seems to be caused by adding a Value type to an int type.

I will go ahead and fix this, but if preferred I can submit a smaller PR tonight which enforce that all TransactionOutput.amount are of type Value (as briefly discussed on Discord).
Briefly trying it out I saw that all tests pass except for one (test_not_enough_input_amount) so I hope it might not be too complex to include already.

but if preferred I can submit a smaller PR tonight which enforce that all TransactionOutput.amount are of type Value (as briefly discussed on Discord).

Thanks for the update @astrojarred ! Feel free to submit another PR for it. I can do that later too if you are too busy in fixing this PR. Really appreciate your helps!

Oh dang, thought I linted this alright. Will merge in the new PR and push again anyway :)

Thank you @astrojarred! All tests are passing now, that is awesome! I will try adding some unit tests before merging.

That's great, I'm so glad! I was also just thinking now about tests. One I thought of is:

sending an entire UTxO in a single output (fee gets subtracted from UTxO) by adding no outputs and just specifying a change address


Audit command line options

Should they all be there? Which should be considered stable vs unstable? Are the arguments the best they should be? Are we missing any?

Rustfmt options:
usage: target/debug/rustfmt [options] <file>...

Options:
        --color [always|never|auto]
                        Use colored output (if supported)
        --config-help   Show details of rustfmt configuration options
        --config-path [Path for the configuration file]
                        Recursively searches the given path for the
                        rustfmt.toml config file. If not found reverts to the
                        input file path
        --dump-default-config [PATH]
                        Dumps default configuration to PATH. PATH defaults to
                        stdout, if omitted.
        --dump-minimal-config PATH
                        Dumps configuration options that were checked during
                        formatting to a file.
        --error-on-unformatted
                        Error if unable to get comments or string literals
                        within max_width, or they are left with trailing
                        whitespaces
        --file-lines JSON
                        Format specified line ranges. See README for more
                        detail on the JSON format.
    -h, --help          Show this message
        --skip-children
                        Don't reformat child modules
        --unstable-features
                        Enables unstable features. Only available on nightly
                        channel
    -v, --verbose       Print verbose output
    -V, --version       Show version information
        --write-mode [replace|overwrite|display|plain|diff|coverage|checkstyle]
                        How to write output (not usable when piping from
                        stdin)

cargo fmt options:
usage: cargo fmt [options]

Options:
    -h, --help          show this message
    -q, --quiet         no output printed to stdout
    -v, --verbose       use verbose output
    -p, --package <package>
                        specify package to format (only usable in workspaces)
        --version       print rustfmt version and exit
        --all           format all packages (only usable in workspaces)

This utility formats all bin and lib files of the current crate using rustfmt. Arguments after `--` are passed to rustfmt.

Looking at cargo fmt first.
I think having a minimal set of Cargo-like options and then using -- for the Rustfmt options is a good strategy. Under that model, my only query is whether we should support --version, since -- -V would work and be less confusing with cargo --version.
An alternative would be to support Rustfmt options without the --. cargo build, for example, supports a lot more options and does so directly, however, cargo rustc supports the -- for passing arguments to rustc. I guess it depends whether we think the mental model should be that cargo fmt is a thing or a shim around rustfmt.

We don't currently have unstable command line options, but I think we should and I think the following should be unstable: --file-lines and --dump-minimal-config, and write modes plain|diff|coverage|checkstyle. I am unsure about --dump-default-config - it seems useful, but I'm not sure about any back compat hazard from using the generated config or if it takes account of unstable features, etc.
Otherwise I think the command line flags looks good. I think for both binaries, we should consider the ordering of the flags in the help message.

And just to be really organised, a TODO list:

[ ] implement unstable command line flags
[ ] check the correct flags and write modes are unstable
[ ] check ordering of flags in help messages (and that unstable ones are only shown as appropriate)
[ ] decide on -- for cargo fmt
[ ] decide on --version for cargo fmt

Proposal for a new CLI, inspired by #1028. The help messages need some work,
OPTIONS:
    --check
                               Check if formatting is correct, emit diff if not
    --emit [files|stdout|checkstyle|coverage]
                               What to emit
    --backup
                               Backup any files which are changed
    --color [always|never|auto]
                               Use colored output (if supported)
    --config-path [PATH]
                               Recursively searches the given path for the
                               rustfmt.toml config file. If not found reverts to the
                               input file path
    --print-config [minimal|default] [PATH]
                               Dumps default or minimal configuration to PATH. PATH defaults to
                               stdout, if omitted. A minimal config is the options checked
                               during formatting.
    --file-lines JSON
                               Format specified line ranges. See `--help file-lines` for details
    --unstable-features
                               Enables unstable features. Only available on nightly
                               channel
    -v, --verbose              Print verbose output
    --quiet                    Emit less info
    -h, --help [topic]         Prints help information
                               Displays detailed help for TOPIC [values: config, file-lines]
    -V, --version              Prints version information

Questions:

--print-config or --print
--error-on-unformatted - do we need it? Feels like a debugging option (and can be used via config file)
--skip-children - feels like it is better used via a config file (or API)
do we need --emit diff? --check basically does this
what should be unstable?

We could remove --emit files since that is the default behaviour.

On --error-on-unformatted:
In order to check formatting on CI, there needs to be an option that returns a failing error code when the source is not formatted. If this is how --check behaves, then --error-on-unformatted would indeed be redundant.

We could remove --emit files since that is the default behaviour.

It's generally considered good practice to support defaults as options in case someone wants to incorporate a tool into a frontend or automation harness, so they can just specify --emit {} rather than needing a special branch just to omit the --emit entirely if the default is chosen.
Ideally, you also support multiple copies of the same option in a "last one wins" configuration so it's possible for things like the alias shell built-in to handle custom defaults simply by appending arguments to a pre-specified list of them.


Needs much better documentation

The current instructions in the README.md simply say to add the following to your build.gradle file:
plugins {
  id "us.kirchmeier.capsule" version "1.0-rc1"
}

task fatCapsule(type: FatCapsule) {
  applicationClass 'com.foo.HelloWorld'
}

But having tried that, these instructions will clearly not work for everybody.  First of all, it doesn't appear that plugins configuration block works like this in Gradle 1.12 (what I was using).  Second, there are Gradle rules that restrict where plugins may appear.
So basically, I followed the instructions and it didn't work.  And the instructions don't tell me what Gradle version I need in the first place.

Fair enough. I wrote in an assumption that people were familiar with Gradle. Given it's complexity, I'm not sure that was a fair expectation to have.
I'll add back in some commentary regarding which Gradle version to use and how to use older versions.

@ldcasillas-progreso I tested the plugin with various versions of gradle and found that the plugin requires gradle 2.0 and above. This finding and other instructions have been added to the README.

Thanks!  Yeah, it's better now.


md-tooltip breaks with vue-router

Steps to reproduce
Codepen: http://codepen.io/anon/pen/pNmpEK
Click button to switch route.
Observe error in console and see application no longer works (clicking or md-tooltip.)
Which browser?
Chrome/Firefox
Vue 2.1.6 (2.1.7 just released, but it looks like vue-material may have to be rebuilt to work with it)
Vue Material 0.5.1
Vue Router 2.1.1
What is expected?
md-tooltip not to break :)
What is actually happening?
You should be able to click on the button and switch between route1 and route2, each with their own tooltips. The tooltip runs at load, but after clicking the button and switching routes, the tooltip breaks.
This only seems to happen when switching from a route with a tooltip to a route with a tooltip. The console logs an error which comes down to _this2.$el.parentNode is null when trying to remove a tooltip.
I've tried digging some, but haven't really been able to come up with a solution. Now that I've got a simple reproducible problem, I'll try another stab at finding out what is really wrong.
Thanks,
Kory

Okay, so good news is this is already fixed. I'm now realizing that when I was looking through the commit history that your merges from the dev branch were hidden behind the [...] so I didn't see the commit about fixing this issue.
I also see that there was already an issue (#188), which my issue appears to be an exact duplicate of.
I have rebuilt master and I don't have this issue on that version, so your next build should fix this. For now I'm just running the latest build and sticking it in node_modules.
Thanks for project. This is my first time using it (and Vue), and the experience has been very nice!


Lost modulate in custom shader's texture function.

Godot version
Godot 4 alpha 3
System information
Windows 11, Vulkan, nvidia T600 driver 472.39
Issue description
In gdshader 'texture(TEXTURE, UV)' output only color from origin texture without modulate.
This is diffrent from CanvasItem Material shader, and diffrent from godot 3 too.
Steps to reproduce

create a sprite.
attach a new shader material to the sprite.
create a new visual shader to the material.
create a Texture2D node in visual shader editor.
connect Texture2D node to Output node.
change node modulate color, nothing happen.

Minimal reproduction project
No response

@harrisyu Please upload a minimal reproduction project to make this easier to troubleshoot.

'Minimal reproduction project' was added.

although this examples uses visual shaders, it's probably duplicate of #49781

Duplicate of https://github.com/godotengine/godot/issues/49781.


Incorrect spacial_dim in AttentionPool2d?

Hello,
Thanks for the great work.
I was testing out your model and I noticed that the spacial dim for the AttentionPool2d was set to 313 in here
However, when I tested with the default configurations for the ResNets backbone that you provided, the code only works for when the spacial dim is 63. Is this intended?
Additionally, when I checked the res18_best_ckpt.pth file for the AttentionPool2d layer I saw that the shape was indeed matching the 313 configuration: Layer: att_pool_head.positional_embedding, Shape: torch.Size([1, 314, 256])
I was wondering if you could provide the code in which the 313 AttentionPool2d is used for the best results?
Thank you.

I believe this was a silly mistake on my part where I had the incorrect input dimension. closing this for now! Thanks.


OSGi metadata needs tweaking

I've been trying to find an open-source mock HTTP(S) server for Java that will work in an OSGi environment. So far I have not found one. It seems to me that mockwebserver is the first project of its kind to make an attempt.
I realise that the current version is a rewrite and still in alpha. But I can see that there are some problems with the OSGi metadata as it stands:

The bundles don't export anything. In order to be useful in OSGi, they'll need to export the public API so that OSGi-based test cases can import the packages that they need.
Some classes are in "internal" packages that probably shouldn't be. An example is MockWebServerExtension in the mockwebserver3-junit5 bundle, which is in the package mockwebserver3.junit5.internal. In order to be useful to Jupiter users it needs to be part of the public API and shouldn't be internal.

@kriegfrj Can you put up a PR that addresses those and we can work through it.  These are exactly the sorts of issues that we need to work through before 5.0. Thanks for the feedback, it's really timely.
The okhttp maintainers lack the motivating use cases and experience to get it exactly right, so you are welcome to help us here.

Thank you for your response @yschimke .
I would love to be able to help. Unfortunately, unless I have a direct and immediate need for the functionality for our own internal projects, I am unlikely to be able to find the time for it. I am looking for the path of least resistance, and at the moment I'm pursing another path with slightly less resistance. But if I hit a blockage on the other path, then this will probably be the path of next-least resistance and I'll end up back here. :smile:

Likely we won't fix without someone submitting a PR and testing it. Hard to try to support it without someone with skin in the game.


website is down or moved?

http://mozillaopennews.org seems to still be owned by Mozilla (according to whois), and there's a DNS record for it, but the site is down. Is http://opennews.org supposed to be the new location and it's just missing a redirect?
$ whois mozillaopennews.org
Domain Name: MOZILLAOPENNEWS.ORG
Domain ID: D164462254-LROR
WHOIS Server:
Referral URL: http://www.markmonitor.com
Updated Date: 2015-12-19T10:03:25Z
Creation Date: 2012-01-19T13:02:12Z
Registry Expiry Date: 2018-01-19T13:02:12Z
Sponsoring Registrar: MarkMonitor Inc.
Sponsoring Registrar IANA ID: 292
Domain Status: clientDeleteProhibited https://icann.org/epp#clientDeleteProhibited
Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited
Domain Status: clientUpdateProhibited https://icann.org/epp#clientUpdateProhibited
Registrant ID: mmr-33682
Registrant Name: DNS Admin
Registrant Organization: Mozilla Corporation
Registrant Street: 331 E. Evelyn Ave
Registrant City: Mountain View
Registrant State/Province: CA
Registrant Postal Code: 94041
Registrant Country: US
Registrant Phone: +1.6509030800
Registrant Phone Ext:
Registrant Fax:
Registrant Fax Ext:
Registrant Email<EMAIL_ADDRESS>Admin ID: mmr-33682
Admin Name: DNS Admin
Admin Organization: Mozilla Corporation
Admin Street: 331 E. Evelyn Ave
Admin City: Mountain View
Admin State/Province: CA
Admin Postal Code: 94041
Admin Country: US
Admin Phone: +1.6509030800
Admin Phone Ext:
Admin Fax:
Admin Fax Ext:
Admin Email<EMAIL_ADDRESS>Tech ID: mmr-33682
Tech Name: DNS Admin
Tech Organization: Mozilla Corporation
Tech Street: 331 E. Evelyn Ave
Tech City: Mountain View
Tech State/Province: CA
Tech Postal Code: 94041
Tech Country: US
Tech Phone: +1.6509030800
Tech Phone Ext:
Tech Fax:
Tech Fax Ext:
Tech Email<EMAIL_ADDRESS>Name Server: NS-1359.AWSDNS-41.ORG
Name Server: NS-1627.AWSDNS-11.CO.UK
Name Server: NS-667.AWSDNS-19.NET
Name Server: NS-397.AWSDNS-49.COM
DNSSEC: unsigned
>>> Last update of WHOIS database: 2016-09-02T13:08:29Z <<<

$ dig mozillaopennews.org

; <<>> DiG 9.10.4-P2 <<>> mozillaopennews.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26822
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;mozillaopennews.org.		IN	A

;; AUTHORITY SECTION:
mozillaopennews.org.	899	IN	SOA	ns-1359.awsdns-41.org. awsdns-hostmaster.amazon.com. 1 7200<PHONE_NUMBER> 86400

;; Query time: 175 msec
;; SERVER: <IP_ADDRESS>#53(<IP_ADDRESS>)
;; WHEN: Fri Sep 02 09:17:50 EDT 2016
;; MSG SIZE  rcvd: 130

Yes! A redirect would be perfect.


[FEATURE] NPC Graphics & Animation

Overview
Animate NPCs with walking, gathering, and idle behaviors, using texture switching for different actions and states.
Goals

[ ] Animate NPC walking, gathering, and idle states.
[ ] Implement texture switching for various actions.

Testing

[ ] Verify smooth transitions between animations for different states.
[ ] Ensure consistency in character animations across different NPC actions.

We don't need that, not for this kind of the project.


Redirecting output to a file misses time output / CPU benchmarking

The CPU benchmarking values measured using time are not redirected to the file appropriately, since time prints to stderr by default. I.e. when running bash nensh.sh > nensh.txt the times will be printed to the console (stderr) and not show up in nensh.txt
Changing the lines
    time "$gnu_dd" if=/dev/zero bs=1M count=500 2> /dev/null | \
        "$@" > /dev/null

to
    ( time "$gnu_dd" if=/dev/zero bs=1M count=500 2> /dev/null | \
        "$@" > /dev/null ) 2>&1

Fixes the issue. There may be an easier solution to this, I'm not well versed in shell programming.

I just noticed this myself. This  little bash script is useful. It would be even more useful if this issue was resolved.

@saua Fixed at last.
Thank you for your input and sorry for the delay — I was busy when you first reported the issue, and it just got lost in ye olde to-do list afterwards. :/


⚠ Bofrost.at has degraded performance

In cb20e74, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1124 ms

Resolved: Bofrost.at performance has improved in 26aab7b.


Pass initialValue prop into Field's component

Feature request

Redux-form 6.5.0

Currently, if a field has initialValue in store and a custom component is used, it's not possible to access this value from it. It seems that initialValue is resolved here but is not passed into the component.
It would be great if it was passed in input or meta prop so that it would be easy to reset the value from custom component.

@erikras maybe you can have a look at this, is there some kind of workaround for now?

Seems to be resolved with https://github.com/erikras/redux-form/pull/2858


Adding logs for "waitForAllDocsLoaded"

Some flaky tests time out whiling waiting for all docs loaded.
This pr adds more information to logs.

Codecov Report

Merging #4729 into master will decrease coverage by 19.43%.
The diff coverage is n/a.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 860cb92...eec2c77. Read the comment docs.


Push consumer with concurrent message listener

Hi, this repo is archived.
You can move to rocketmq-clients for more details.


Remote repetitive structures and halos cause near-range noise

Required Info |
-- | --
Camera Model | D435
Firmware Version | <IP_ADDRESS>
Operating System & Version | Ubuntu
Platform | RK3399
SDK Version | 2.18.1
Language | c++
Segment | Robot
D435 is in high precision mode， the resolution is 848x480. We use it to avoid obstacles in the restaurant.But the near-range noise will make the robot stop!
The closer the red pixel is, the closer it is.

(The repetitive structures cause near-range noise)

(The halos cause near-range noise)

(The repetitive structures cause near-range noise)
I try to set the API setting “DSSecondPeakThreshold.” ,but it can't completely filter out these noises and make the whole point cloud sparse.
Please tell me how to filter out these noises or how to detect this area, thank you!!!

Could you specify the range of detection you are working with ?
I'm no expert whatsoever, but it might be easier to set a small viewing range (50 cm to 150 cm maybe ?) and to focus on improving this "narrowed" point cloud.
Setting the Depth Units, Depth Clamp Min & Max and adjusting the Disparity Shift under Stereo Module/Advanced Controls/Depth Table will do that.
I'm not sure about what output you are expecting (completely filled mesh ? sharp objects ?) though...
With those cameras, there is always a risk of losing data depending on the environment. Things like office lights or reflective surfaces (like this big fridge in your picture) tends to mess a bit with the reconstruction. Also, as you stated, repetitive patterns are not helping !
-> Setting the exposure on auto could help, try adjusting the laser power too.
Anyway, once you've set your parameters, I would advise you to take a look at the post processing filters, those can really improve the overall quality of your point cloud.
Hope this will help !

Could you specify the range of detection you are working with ?
I'm no expert whatsoever, but it might be easier to set a small viewing range (50 cm to 150 cm maybe ?) and to focus on improving this "narrowed" point cloud.
Setting the Depth Units, Depth Clamp Min & Max and adjusting the Disparity Shift under Stereo Module/Advanced Controls/Depth Table will do that.
I'm not sure about what output you are expecting (completely filled mesh ? sharp objects ?) though...
With those cameras, there is always a risk of losing data depending on the environment. Things like office lights or reflective surfaces (like this big fridge in your picture) tends to mess a bit with the reconstruction. Also, as you stated, repetitive patterns are not helping !
-> Setting the exposure on auto could help, try adjusting the laser power too.
Anyway, once you've set your parameters, I would advise you to take a look at the post processing filters, those can really improve the overall quality of your point cloud.
Hope this will help !

0-3 meter I am working with

One solution that tested well in the field is adding a physical filter in front of the camera, that would pass IR and block visible light, and increasing the laser power.
At close range it would break repetitive patterns by introducing a little bit of texture from the projector, and at longer range it will filter the false alarms because less light from that area will make it into the camera. This has proven to work fairly well for customers operating indoor for collision avoidance use case.
We also tried different software based solutions, but it's hard to filter the outliers without significantly reducing the total number of points.

One solution that tested well in the field is adding a physical filter in front of the camera, that would pass IR and block visible light, and increasing the laser power.
At close range it would break repetitive patterns by introducing a little bit of texture from the projector, and at longer range it will filter the false alarms because less light from that area will make it into the camera. This has proven to work fairly well for customers operating indoor for collision avoidance use case.
We also tried different software based solutions, but it's hard to filter the outliers without significantly reducing the total number of points.

Do you have an adapted physical filter recommendation?Or how can I find this physical filter?

I can't find some  adapted physical filter.Do you have an adapted physical filter recommendation?Or how can I find this physical filter?Thank you!!!

Haven't personally tested it, but I think that you are looking for something like the followings :
Either a long-pass filter, which will let pass anything around 750 nm and beyond.
Or a band-pass filter, which will let pass a precise range of wavelengths.
In any case, you'll have to aim for a 850 nm wavelength (the IR emitter) for your choices.

Hi McJun1,
Do you find the filter? You may try filter Lee #87.


Improved definitions for cosmic, etc

This tweaks the definitions for cosmic, $\sigma$-space, $\aleph_0$-space, $\aleph$-space.  In particular, I thought the reference that was used was not very good.  I replaced it with the original paper that introduced the concept in each case, plus a good recent paper (Gabriyelyan & Kakol) that has all the involved notions (and more) in a well organized manner.  Also tweaked the definitions of network to follow Engelking.
Regarding the fact that all four definitions assume $T_3$, they were initially defined that way, except for $\sigma$-space.  But it seems recent papers usually assume $T_3$ also.
In particular, [cosmic <==> T3 + countable network].  But in order to derive basic implications, it seems to me it would be useful to add the property "has a countable network" by itself.  That would allow to derive simple things like [countable ==> has a countable network], etc without having to repeat T3 (countable + T3 ==> cosmic) for example.
@creator1creeper1, @StevenClontz, what do you think?

@creator1creeper1 fyi, you can see how this PR would look like by clicking on the Advanced tab of the pi-base web page, entering the branch name (in this case "cosmic-defs") in the data branch field and clicking Sync.

I'd be in favor of adding a property like "has a countable network" and theorems like "second countable => has a countable network", to avoid invoking $T_3$ except where necessary.

I'd be in favor of adding a property like "has a countable network" and theorems like "second countable => has a countable network", to avoid invoking

Sounds good.  I'll add this to comments for #511.


Bond.JSON silently references Newtonsoft.Json v9

Bond.JSON silently references Newtonsoft.Json v9 with no listed dependencies in nuget feed.

Can you be more specific about the package you are referring to? There's no official Bond package named "Bond.JSON". Bond.Runtime.CSharp (.nuspec/NuGet.org listing) is where the JSON serialization code lives. It lists Newtonsoft.Json >= 9.0.1 as a dependency.

My bad didn't realized that it referenced runtime package.


Deactivated account gone

After I deactivated an email account in Dynamics365 | Settings | Email Configuration, I can't see the email account from either All Mailboxes or Inactive Mailboxes. Not sure why?
update: After quite some delay, the deactivated accounts appeared under Inactive.
Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 6a82fbba-3f7c-555f-1bb6-62adf4680efd
Version Independent ID: 616e8c25-2f9f-a15b-7baa-9ea74819d513
Content: Deactivate accounts or contacts in Dynamics 365 Customer Engagement (on-premises)
Content Source: ce/customerengagement/on-premises/basics/deactivate-activate-account-contact.md
Product: d365ce-op
GitHub Login: @mduelae
Microsoft Alias: anbichse

After quite some delay, the deactivated accounts appeared under Inactive Mailboxes.


Replace tomcat7-maven-plugin with jetty-maven-plugin

Problem description
After struggling to verify the workbench in https://github.com/eclipse/rdf4j/pull/4375 using Docker, I tried using tomcat7-maven-plugin, which is configured in the server module. But I realized that this plugin had its latest release in 2013 and seems abandoned.
For this reason, I am suggesting migrating to jetty-maven-plugin, which is much better documented and seems to work pretty well. I actually have prepared a branch, and will submit a PR for review right after getting the issue number. 😉
Preferred solution
No response
Are you interested in contributing a solution yourself?
Yes
Alternatives you've considered
No response
Anything else?
No response

Excellent idea, thanks !

Excellent idea, thanks !

Thanks @barthanssens! Please review https://github.com/eclipse/rdf4j/pull/4447.


how to reconstruct the original background in the diffuser-based version

hello, excellent work. but when i edit a picture in the diffuser-based version, the background changed a little, so how can i reconstruct the original background in the diffuser-based version.

Same question


Automated cherry pick of #10499: vpcagent: ovn: fix qos for eip upload

Cherry pick of #10499 on release/3.3.
#10499: vpcagent: ovn: fix qos for eip upload

Codecov Report

Merging #10504 (10909a4) into release/3.3 (9a6c26d) will decrease coverage by 2.70%.
The diff coverage is 0.29%.

@@               Coverage Diff               @@
##           release/3.3   #10504      +/-   ##
===============================================
- Coverage         6.78%    4.08%   -2.71%
===============================================
  Files              676      281     -395
  Lines           118955    59502   -59453
===============================================
- Hits              8076     2431    -5645
+ Misses          110267    56878   -53389
+ Partials           612      193     -419

Flag
Coverage Δ

aFlag
4.08% <0.29%> (-2.71%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

pkg/cloudcommon/notifyclient/notify.go
2.22% <0.00%> (-0.36%)
:arrow_down:

pkg/compute/models/buckets.go
1.10% <0.00%> (-0.07%)
:arrow_down:

pkg/compute/models/capabilities.go
0.00% <0.00%> (ø)

pkg/compute/models/cloudproviderregions.go
0.00% <0.00%> (-0.27%)
:arrow_down:

pkg/compute/models/cloudproviders.go
0.87% <0.00%> (+0.01%)
:arrow_up:

pkg/compute/models/cloudregions.go
1.54% <0.00%> (-0.09%)
:arrow_down:

pkg/compute/models/cloudsync.go
0.00% <0.00%> (ø)

pkg/compute/models/dbinstance_accounts.go
2.05% <0.00%> (-0.03%)
:arrow_down:

pkg/compute/models/dbinstance_backups.go
2.78% <0.00%> (-0.17%)
:arrow_down:

pkg/compute/models/dbinstance_databases.go
3.33% <0.00%> (-0.07%)
:arrow_down:

... and 709 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0c6d954...10909a4. Read the comment docs.


Logging fixes

Fleshed out some details with the cli/logging including creating an output directory if one doesn't exist and naming logs based on when the program is executed instead of just appending to an existing log.

using logging_redirect_tqdm actually redirects to stdout when trying to write to a file, working on a fix

Added those explanations to the code so it's clear for everyone, should be all set to merge afterwards unless there are any last changes.


Feature request: rename columns when selecting

Since xsv is so invaluable for splitting and reformatting big messy CSV files (thanks!), it'd be really useful to be able to rename columns when select-ing them out: I deal with some crappy CSV files that have 150-character header names.
At first I was thinking a special syntax for select might be the solution, but now I suspect a rename subcommand might be the more appropriate option. Not really up to speed with Rust dev yet, so starting off by sharing this idea for feedback.

This sounds like a possibly good idea to me. It probably should be a separate rename command. It will need some design work in terms of the syntax supported. It would ideally combine select syntax for specifying which columns to rename, for example.

Yeah, I'm thinking something like 1=New name,Code=value,Dupe[3]="Nondupe","Quoted[2]"=something. There's no need for the 1-4 column range syntax, or ! exclusion for a pure rename command, which makes this simpler than cramming renaming into select.

A simpler version might be too just keep the same select syntax, which selects some number of columns in a particular order. Then the remaining positional arguments are the new names of the columns, where there are exactly N arguments, where N is the number of fields selected.

Ah, yes. I can see that might work. The simple case would then just be xsv rename "Foo bar" "Fubar", which is nice.
I had a look through past issues etc., and what we're talking about here sounds a bit like the --set-headers suggestion which was part of #96, but a bit cleaner IMO.

+1 for this feature! Definitely have had the need for this a few times.

@BurntSushi @purcell @kacrouse I had to merge/join CSV files today, and I just stumbled upon xsv and it's a really neat project!
I really needed to be able to rename columns so I tried to figure out what would be the best way to contribute a solution. I implemented what was discussed in this thread, specifically the variant suggested by @BurntSushi. Let me know if this hits the spot!

a sql like syntax for this so something like xsv select "personName as name,user_id as id" users.csv

xsv is never going to get a SQL-like syntax. If you want SQL, load the data into SQLite or some other relation database.

I appreciate that you don't want to support the full sql syntax. I already use csvsql (comes with csvkit), which indeed uses sqlite. My main issue with that is that it is kind of slow compared to xsv, which is awesome.
The reason I proposed this AS syntax is because it is an elegant solution with minimal impact that is easy to understand and probably also to implement and pretty.