add grafana & fix some bug

How about rename to `apisix_http_prometheus.json'?
compose/grafana_conf/dashboards/apisix.json

yes, we need the asf header
琚致远<EMAIL_ADDRESS>于 2020年6月12日周五 下午9:16写道：

@juzhiyuan requested changes on this pull request.
In compose/grafana_conf/provisioning/dashboards/all.yaml
https://github.com/apache/incubator-apisix-dashboard/pull/257#discussion_r439411241
:

@@ -0,0 +1,11 @@
+apiVersion: 1

+providers:
+- name: 'default'

orgId: 1
folder: ''
type: file
disableDeletion: false
editable: false
options:
path: /var/lib/grafana/dashboards

It should be dashboard
In api/build.sh
https://github.com/apache/incubator-apisix-dashboard/pull/257#discussion_r439411495
:

@@ -1,20 +1,3 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more

@moonming https://github.com/moonming We may still need this header,
right?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/incubator-apisix-dashboard/pull/257#pullrequestreview-429730171,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AGJZBK2GNTQ2JBVCA7BYYRTRWITCLANCNFSM4N4G4IRQ
.

Any updates?

apisix_http_prometheus

ok

How about rename to `apisix_http_prometheus.json'?
compose/grafana_conf/dashboards/apisix.json

done


Enhancement: Edit prefixes for Add / Edit command

Currently bugged.
Failing the AddCommandSystemTest and EditCommandSystemTest now.

Branch deleted, PR closed.


Make GetFeature-Urls ending with & work, change logger to actually use "owslib" namespace

This fixes a bug where a GetFeature URL ending with "&" was incorrectly constructed.
Also while debugging I was missing logs in the "owslib" namespace (as written in the docs). I think the second change in this MR fixes this.

@lordi thanks for the valuable contribution.  It turns out that we log as logging.getLogger(__name__) in the library (though we say otherwise in the docs, README and tests).  Can you update this PR set the logger to __name__?  I'll update the docs/test/README in this regard for consistency.  Thanks.

I saw that master removed owslib.util.log in the mean time, so I rebased my PR to only include the amp change.

Closing as should be fixed in 0.32.0 release


Spring boot with redisson 3.3.2 fails without optional actuator dependency

Actuator is an optional feature in Spring Boot, there are a lot of projects that do not use actuator due to additional dependencies and complexity.
Redisson 3.3.1 worked fine without actuator but 3.3.2 fails to start with ClassNotFoundException for  org.springframework.boot.actuate.cache.CacheStatisticsProvider.
Suggested change registers Redisson AutoConfiguration only if actuator is in the classpath

I did not see previously that master branch also has the same issue and same fix needs to be applied there

@usultis Please apply your changes against master only, it will be merged with 3.0.0 before release

@mrniko  corrected

Thanks


feat: Show volume of AR that has gone through Verto

@maximousblk @martonlederer Would love some help with styling 😁

Will check it in a second :)


Add fixes from review

Models

Comprehensively rename Entity and Column to Table and Field, respectively
Rename model attributes to be clear about which service the are connected to

Templates

Add meaningful titles for each page in title block
Increased bottom margin on main
Removed Bootstrap JS
Added h1 to Export Job form page
Added spacing between form inputs
Reimplemented checkboxes as fieldset
Made cancel delete button orange
Move inline styles to CSS class

Forms

Added help text for export location
Debugged setting of formset prefixes

Tests

Made all individuals in test fixtures fake
Updated form data fixtures

@HaSistrunk Thank you so much!


error importing ChartistModule

using ChartistComponent is working fine but if I try to import ChartistModule, there's error on console. I have to use chartist in multiple modules so can't import the component.

I'll take a look, thanks for the report.

its because ChartComponent is not exported from module. Adding
exports: [
    ChartistComponent
  ]

to chartist.module fixes this.

Great thank you! I'll probably fix this tonight, feel free to submit a PR if you need it urgently though.

This should be available on 0.9.2. Thanks for the report.


Datepicker Timezone

I don't know if I'm just missing something but I think currently the datepicker will always generate a date in the browser timezone. This causes a problem for me because I would like to use UTC dates instead. I thought I could just fix the date object on (change) or something but this appears to break the selected display date logic and the selected date will move to the next day. It would be awesome if you could tell the date picker to generate dates in UTC instead of the default timezone or if you could suggest a workaround.
Many thanks.

+1


Images using AWS link

Hi! I love your plug-in.  we are using it b/c it provides the perfect tool between a full CMS but still allowing editing.
I added my images to the blog from AWS, but seems the bootsy tool keeps converting it back to a locally stored image.  I don't want to installed carrierwave gem b/c we are using paperclip throughout the site and I'm worried about conflicts.
Is there anyway to tell bootsy to just leave it hosted on AWS and not convert it to a locally sourced file?
thanks

Hi there! Thanks for using Bootsy.
If you don't want to store the images using the editor and just reference them via URL you can do so by disabling the upload feature: https://github.com/volmer/bootsy#uploads

Thanks for your quick reply.
Just to be clear, we are only using AWS links but the bootsy editor is converting them to local files.
Also, can you give an example of what this means?  It's not clear to me:
Uploads

If you don't want to have image uploads, just call bootsy_area in a form builder not associated to a Bootsy::Container model. This way users will still be able to insert images in the text area using an external image URL.

I'm currently using bootsy for my blog pages.  Here is my code - how do I change the code to address your suggestion?
<%= simple_form_for(@blog) do |f| %>
  <%= f.error_notification %>

    <%= f.input :image, label: "Blog image", :as => :file %>
    <%= f.input :title %>
    <%= f.input :content, as: :bootsy, :input_html => { :rows => 50} %>
    <%= f.input :publish, as: :boolean, label: "Do you want to publish this blog?" %>

    <%= f.button :submit, class: 'btn btn-outline p-t-10 p-b-10 m-b-10' %>

<% end %>

How would I change this to address your suggestion?
thanks again for the help

Hi
So I was able to change my form to the following:
<%= simple_form_for(@blog) do |f| %>
  <%= f.error_notification %>

    <%= f.input :image, label: "Blog image", :as => :file %>
    <%= f.input :title %>
    <%= f.input :publish_date %>
    <%= f.input :content, as: :bootsy, editor_options: { image: false, html: true }, :input_html => { :rows => 50} %>
    <%= f.input :publish, as: :boolean, label: "Do you want to publish this blog?" %>

    <%= f.button :submit, class: 'btn btn-outline p-t-10 p-b-10 m-b-10' %>

<% end %>

And I removed from blog.rb
include Bootsy::Container
I can then insert an image into the Bootsy HTML editor.  However this is a very clunky solution. When working with the HTML it doesn't accept pasted input very well and has limited options.
Can you please tell me if there is a way I can continue to use the image tool, insert a link, and have it NOT change it to a bootsy saved file. But instead keep it as an AWS link.
thanks for your help1

how do I change the code to address your suggestion?

You just need to remove include Bootsy::Container from your Blog model. That way you would still be able to insert images in the editor, but with URLs instead of uploads. You will have essentially this modal:

Got it -I did that as well but it doesn't allow then for the same editing functionality (essentially allowing the text to wrap around the image)
Is there anyway to disable bootsy from changing the default url to /uploads/bootsy/image/...
and just get it to leave in the AWS URL provided?
thanks

Unfortunately not. The image positioning options are only available to uploaded images for now.

Ok great - thanks for your help.  I love your tool so I'm hoping to find a good solution for this.


Generate Sitemap.xml and Robots.txt to improve SEO and Site Crawling

Adding sitemap.xml and robots.txt files helps optimize a website for search engines.
Sitemap.xml provides a list of important URLs, helping search engines discover, crawl, and index new and updated content faster, improving SEO.
Robots.txt controls which parts of the site search engines can access, preventing indexing of sensitive or unimportant pages, conserving crawl budget, and reducing server load.
Together, they enhance site visibility, performance, and search engine ranking.
ASSIGN WITH LEVEL 2 OR LEVEL 3

Hey @amiya-cyber , We don't need this feature right now, but thanks for the suggestion.


Add "every day" at "Charging Plan" "Departure" "Day"

May I kindly ask that you add "every day" in the key word list of the field "Day" in the Departure Charging Plan dialogue.
Background: Charging at Arrival typically leads to the situation that you charge during the evening, after you came back from work. This typically is the time of the highest demand in the grid. This could be avoided if instead of using planned charging at arrival, users could simply choose every day at 5:00 am the charging goal of xx % should be achieved.
Thanks in advance

Thats the repeating plan.


Create link_sharepoint_folder_file.yml

Description
Add coverage for sharepoint folder shares which contain a single suspicious file.
Associated samples

Sample 1
Sample 2

Associated hunts

Hunt 1

/update-test-rules

LA in the rule means this won't go into test rules, will leave deployed on shared samples for awhile and see how it goes

low volume in shared samples. logic is pretty tight.  ready for review


[Discussion] - Opposing candidates.txt for maintainers that don't want their STUN servers tracked like this?

The README contains a section about being able to remove stun servers from parties that don't want their servers listed here - but is there a tracking mechanism to prevent them from being added by someone else in the future again?
Should there be a file similar to candidates.txt that lists out servers that have specifically asked to not be a part of this project? Or is there some other, better, way that you can track which servers should be rejected from new issues/PRs?

Hi, thanks for this.
When someone comes and asks for their service to be removed (so far it's happened once), I just comment out the URL in the candidates.txt file. This way the service is excluded from the checks (and it never appears on valid_* lists), and at the same time we remember to never add it again.
If someone complained that their service should be completely removed from this repo, then I would probably tag their issue with a new tag, and then for every new service, we would have to look through all previous removal requests first, as identified with that new tag. Luckily this hasn't happened yet.

If you have ideas on how to approach this better, then let me know, otherwise I'll close the issue


Is this still actively maintained?

@esvit - are you still maintaining ngTable?

Same question here

Hello, it will be actively maintained. New version will be probably in a month

cc: @AhmedSamy

@iyel did you overtake?
cheers


Exception in GET /mship/auth/verify

Error in VATSIM-UK/Core (Laravel)
Exception in GET /mship/auth/verify
Unable to communicate securely with peer: requested domain name does not match the server's certificate.
View on Bugsnag
Stacktrace
/var/www/core/releases/1484327451/app/Http/Controllers/Mship/Authentication.php:272 - App\Http\Controllers\Mship\Authentication::App\Http\Controllers\Mship\{closure}

View full stacktrace

Any additional ideas on this error, @NFarrington / @atoff ? There's got to be a root cause, it's been happening on/off for a few months now.

Could it be anything to do with the domain change? Same kind of thing as #589

They're the same error, essentially.  It's been happening since before the domain changed.  @khardern can I just check with you that there isn't any form of rate limiting/redirecting in place for a high number of requested?


empty space in PHP file generate error

https://github.com/TeamEver/everblock/blob/4dc25cd7c4e7776d567c26a4a0b399f3631d4c66/controllers/admin/AdminEverBlockController.php#L1

Hello @pawelkierklo
Fix has been pushed on latest release
https://github.com/TeamEver/everblock/releases/tag/V5.8.2


test-branch/execute-job/wrong-commit-sha-on-pr/e145302e-45a7-4a86-9f5f-4f365b9247e4

PR for testing 51cbd4048c499c9565010720a22f0c6b852adc86

/canonical/self-hosted-runners/run-workflows 4f1c70cc3dad5370c365b76d0fa1a3c58185515e


Optimize small integer unpack

https://github.com/neo4j-php/Bolt/blob/1efed4ba3f1796d37d2715998e5411759f580607/src/PackStream/v1/Unpacker.php#L255-L258
You can do it better than two if's. It's just a number between -16 and 127.
https://7687.org/packstream/packstream-specification-1.html#integer

#87


Addition of relevant data flow diagrams on README.md using Mermaid

A data flow diagram detailing the state of data flow between the user, Stargate and Cassandra, by making use of Mermaid.

What diagram types could be useful?

Sequence diagram  https://mermaid-js.github.io/mermaid/#/sequenceDiagram
Relationship diagram https://mermaid-js.github.io/mermaid/#/entityRelationshipDiagram
?


Docs: Update index.rst for installation and test changes

Closes https://github.com/zarr-developers/numcodecs/issues/425
This PR updates the instructions for installation and testing at:
https://numcodecs.readthedocs.io/en/stable/index.html
Specifically, it instructs to use pip and not python setup.py install and to use pytest and not nose.
I also changed the way the DISABLE flags are mentioned. I made the changes based on my attempts using arm64 macOS (so no AVX2, no SSE2). With pip install, setting the env flags wasn't needed, so I mention them as an extra note.
TODO:

[ ] Unit tests and/or doctests in docstrings
[ ] Tests pass locally
[ ] Docstrings and API docs for any new/modified user-facing classes and functions
[ ] Changes documented in docs/release.rst
[x] Docs build locally
[ ] GitHub Actions CI passes
[ ] Test coverage to 100% (Codecov passes)

Workflows launched, cc: @jakirkham

This now matches the other merged PRs. Merging but if there are any issues, we can update the mainline and the docs together. Once again, thanks, @psobolewskiPhD.


Slow response when system is offline

When system is connected to network but don't have access to internet (just local network) navigation, move , copy , rename in directories and files become very slow. it's happening periodically and seems  like it's waiting for kind of timeout.

@moostad when this happens can you open chrome tools (Help > Toggle Developer Tools), open the Profile tab and run a CPU profile to see where the time is spend?

@bpasero I don't see any problem in CPU profile (or most likely I'm not qualified to read it) but I can send it to you if it can help. the only thing I noticed is every time that's happening two error will print in console.
1: Error: getaddrinfo ENOTFOUND vortex.data.microsoft.com vortex.data.microsoft.com:443 at errnoException (dns.js:25:10) at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:76:26)
2: Error: getaddrinfo ENOTFOUND dc.services.visualstudio.com dc.services.visualstudio.com:80 at errnoException (dns.js:25:10) at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:76:26)

btw - this is issue #1000 :tada:

@moostad if you are able to capture a CPU profile when the explorer shows the spinning progress you can send it to me.
I think the ENOTFOUND is from our telemetry reporting. @SofianHn any chance this could slowdown operations? We don't send telemetry sync do we?

@bpasero we don't sent telemetry sync, those errors are expected to be on the console. I just tried to reproduce it with no luck.
@moostad please try disable telemetry as described here and let us know if you see any difference.

@bpasero here's the CPU profile.
CPU-20151204T235311.cpuprofile.zip

@SofianHn I disabled telemetry for about an hour and problem solved and as soon as I enabled that again, I get spinning progress 4 times in less than 10 minutes.

Interesting...thanks @moostad for the quick response. I am guessing you are on Mac, after being offline for a while please run open $TMPDIR/appInsights-node from the terminal and verify what exists in this directory. You should see a bunch of json files, I am interested how many and what's their sizes.
@bpasero I verified that the caching mechanism by AppInsights is async. My guess now that it might be writing to the disk too often which might be causing this issue. I will take it from here.

@SofianHn I will send those information later however I'm not sure that have anything to do with io because I don't get spinning when system is disconnected from network (wifi is off) it's happening only when my mac is connected to the network without internet. I guess when it's see getaway that try to connect to website and block everything until get timeout. that's when I get spinning.
p.s. I have no problem to read and listening English but writing and speaking is another story, so sorry about my poor english.

This time I tried on Ubuntu to be sure it's not OS/Device related, it's happening but not as often and not as long as Mac. maybe because it has much more power then the Mac (2x raid0 SSD for OS, 12 core CPU, 32 GB ram). here's the list of files
Total size: 1.5 MB
Count : 43
  45K Dec  5 11:16<PHONE_NUMBER>361.ai.json
  21K Dec  5 11:16<PHONE_NUMBER>507.ai.json
  44K Dec  5 11:17<PHONE_NUMBER>096.ai.json
 8.0K Dec  5 11:18<PHONE_NUMBER>054.ai.json
  28K Dec  5 11:18<PHONE_NUMBER>515.ai.json
  28K Dec  5 11:18<PHONE_NUMBER>262.ai.json
  14K Dec  5 11:19<PHONE_NUMBER>035.ai.json
  18K Dec  5 11:19<PHONE_NUMBER>832.ai.json
 5.4K Dec  5 11:19<PHONE_NUMBER>224.ai.json
  65K Dec  5 11:20<PHONE_NUMBER>942.ai.json
  54K Dec  5 11:20<PHONE_NUMBER>250.ai.json
  19K Dec  5 11:20<PHONE_NUMBER>519.ai.json
  17K Dec  5 11:21<PHONE_NUMBER>099.ai.json
  26K Dec  5 11:21<PHONE_NUMBER>507.ai.json
  16K Dec  5 11:21<PHONE_NUMBER>055.ai.json
  20K Dec  5 11:22<PHONE_NUMBER>942.ai.json
  18K Dec  5 11:22<PHONE_NUMBER>785.ai.json
  46K Dec  5 11:22<PHONE_NUMBER>527.ai.json
  23K Dec  5 11:23<PHONE_NUMBER>779.ai.json
  43K Dec  5 11:23<PHONE_NUMBER>367.ai.json
  52K Dec  5 11:23<PHONE_NUMBER>358.ai.json
  15K Dec  5 11:24<PHONE_NUMBER>260.ai.json
  46K Dec  5 11:24<PHONE_NUMBER>154.ai.json
  41K Dec  5 11:24<PHONE_NUMBER>161.ai.json
  54K Dec  5 11:25<PHONE_NUMBER>280.ai.json
  37K Dec  5 11:25<PHONE_NUMBER>366.ai.json
  17K Dec  5 11:25<PHONE_NUMBER>179.ai.json
  35K Dec  5 11:25<PHONE_NUMBER>539.ai.json
  34K Dec  5 11:26<PHONE_NUMBER>334.ai.json
 5.4K Dec  5 11:26<PHONE_NUMBER>281.ai.json
  25K Dec  5 11:26<PHONE_NUMBER>101.ai.json
  38K Dec  5 11:27<PHONE_NUMBER>171.ai.json
  25K Dec  5 11:27<PHONE_NUMBER>680.ai.json
  35K Dec  5 11:27<PHONE_NUMBER>804.ai.json
  19K Dec  5 11:28<PHONE_NUMBER>982.ai.json
  66K Dec  5 11:28<PHONE_NUMBER>938.ai.json
  69K Dec  5 11:28<PHONE_NUMBER>167.ai.json
  37K Dec  5 11:28<PHONE_NUMBER>401.ai.json
  31K Dec  5 11:29<PHONE_NUMBER>368.ai.json
 9.5K Dec  5 11:29<PHONE_NUMBER>790.ai.json
 6.8K Dec  5 11:29<PHONE_NUMBER>153.ai.json
  80K Dec  5 11:30<PHONE_NUMBER>946.ai.json
  22K Dec  5 11:30<PHONE_NUMBER>991.ai.json

thanks @moostad for the clarification and the list of files. The update rate and the size of files seems reasonable to me, writing 3 small files a minute should not be a problem. Now that IO is less likely to have caused this, I will work on producing the case with network and without internet to see what might be going wrong here.
You english is great btw, we really appreciate working with us to get this issue fixed.

The electron folks confirmed the issue on their side. Until this issue get fixed in electron and we get the new build, the workaround is to disable telemetry.

We will not update to a new electron build for the December update. Therefore I'm moving the milestone to January.
The work around for how to disable telemetry is document here: https://code.visualstudio.com/Docs/supporting/FAQ#_how-to-disable-telemetry-reporting

Nothing happened in February for this. Ping @SofianHn

I pinged in the upstream, still waiting

We will not update to a new electron for March, shifting to April

This is super super bad, I just hit this while in transit. We have to do something about it for April, it feels you are sitting in front of a web browser that tries to connect to the file that is on disk.

@Microsoft/vscode to get some attention and maybe collect ideas. We could maybe:

take out telemetry from the critical path of opening a file
detect that the system is offline and just not try to connect (Electron has APIs for this)

+2 the telemetry service should not run in the main thread but like search and fs watch in a separate process

as an easy workaround we can use a setTimeout trick to get off the critical path

I find it surprising that every call to telemetry gets send to the appenders right away. I would have at least expected some kind of buffering and batching. Adding some way of buffering with timeouts at the point where we collect these events seems like a good thing to do anyway.
Still, for the actual network access I think we should just turn off network access when we detect we are offline.

I find it surprising that every call to telemetry gets send to the appenders right away.

I think that's fair. The appenders are the ones that know what debouncing strategy to apply. However, the telemetry service should guard itself against evil/slow appenders.

Looking into what application-insights module does it seems they do buffering, however I find it weakly implemented (https://github.com/Microsoft/ApplicationInsights-node.js/blob/master/Library/Channel.ts#L64).
Basically they seem to:

collect events and add them to a buffer
install a timeout of some seconds before sending the buffer
if the buffer has more than 250 items inside, just send directly

Given the DNS lookup might take a long time I think what happens is that the buffer gets filled up to 250 and then it would just get send directly instead of waiting for the other timeout to finish.
Adding our own setTimeout(0) at https://github.com/Microsoft/vscode/blob/master/src/vs/base/node/aiAdapter.ts#L145 seems like one way to get off the critical path of opening an editor.
Thinking more about the issue I have to agree that we need to move telemetry reporting into a separate process because no matter how smart we get, the blocking issue will always be an issue on the renderer side.


ISO 639 "cnr" should be mapped to 'SRB '

Montenegrin (ISO 639 code "cnr") is a variety of Serbian. Since it doesn’t have its own language system tag, "cnr" should be mapped to 'SRB ' (Serbian).

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 662c6f54-f90c-52ed-dca2-0f0a0a49fee5
Version Independent ID: 6aa33924-d45e-60c3-a9fd-e317ff662993
Content: Language system tags - Typography
Content Source: typographydocs/opentype/spec/languagetags.md
Product: typography
GitHub Login: @PeterCon
Microsoft Alias: PeterCon

Hmmm... People that insisted that ISO 639 needed to code "Montenegrin" as distinct from "Serbian" might disagree. Should 'SRB ' be mapped to cnr, or should a separate language system tag be created?

I don’t think it’s necessary, unless there is some evidence that the two varieties have different glyph preferences, though it wouldn’t be wrong to create a new tag. A benefit of mapping it to 'SRB ' is that it is easier to update shaping engines to map "cnr" to 'SRB ' than it is to update all fonts that use 'SRB ' to also use 'CNR '. I leave it to you; my main concern is that it be mapped to something, because otherwise text tagged as Montenegrin will end up using the default (probably Russian) glyphs.

I don't have any strong opinion on this issue beyond that reasonable/relevant alternatives have been considered. Your rationale makes sense to me.
Changing for next version.


Added EVCloudKitDao as app using this library.

EVCloudKitDao is a library for simplified access to CloudKit. As a sample there is a fully functional chat demo included which is using this library

Now including support for sending and receiving pictures.

@evermeer - awesome! :+1: thanks for the update!


BZ-1315671: dev/release branches initially not listed in Repository Editor after managed repository created

@manstis
Hello Mike, this is the PR we chatted about. NOT for merging yet.
Could you please take a look?
Thanks.

@wmedvede It looks fine to me (I assume you'll add a couple of tests)?

@manstis Ok, thanks! Tests are comming up with the mergeable version.

@manstis Hello Mike, the tests has been added. Ready to merge in my opinion.

Thanks @wmedvede
+1 to merge.

6.4.x: f13ebdc190eb49d20fae9ae9665b60ed5611df96


Use prop-types library instead of React.PropTypes

This addresses the following warning in console logs
Accessing PropTypes via the main React package is deprecated

Reference: https://facebook.github.io/react/warnings/dont-call-proptypes.html

Thank you!


[Windows] Race condition with move and directory change2

Commit Message:
There is a weird race condition between MoveFileEx and ReadDirectoryChangeW.
What happens is the following:
Thread 1: MoveFileEx from E:\temp/eds.update.pb_text -> E:\temp/eds.pb_text
Thread 2: ReadDirectoryChangeW notification triggers and notifies filesystem_subscription_impl
Thread 3: filesystem_subscription_impl tries to read file  E:\temp/eds.pb_text and fails
Thread 1:  MoveFileEx completes
These logs also demonstrate it:
❯ ./bazel-bin/test/integration/eds_integration_test.exe --gtest_filter=IpVersions/EdsIntegrationTest.RemoveAfterHcFail/0 -l warn
Note: Google Test filter = IpVersions/EdsIntegrationTest.RemoveAfterHcFail/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from IpVersions/EdsIntegrationTest
[ RUN      ] IpVersions/EdsIntegrationTest.RemoveAfterHcFail/0
[3928][warning][runtime] [source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
[3928][warning][runtime] [source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
[3928][warning][misc] [test/test_common/environment.cc:160] move started
[3928][warning][misc] [test/test_common/environment.cc:163] move completed E:\temp/eds.update.pb_text -> E:\temp/eds.pb_text
[3928][warning][misc] [test/test_common/environment.cc:160] move started
[3928][warning][misc] [test/test_common/environment.cc:163] move completed E:\temp/cds.update.pb_text -> E:\temp/cds.pb_text
[9244][warning][runtime] [source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
[9244][warning][runtime] [source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
[9244][warning][runtime] [source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
[9244][warning][runtime] [source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
[9244][warning][main] [source/server/server.cc:604] there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections
[3928][warning][misc] [test/test_common/environment.cc:160] move started
[1028][warning][misc] [source/common/filesystem/win32/watcher_impl.cc:214] trigger watcher
[9244][warning][misc] [source/common/filesystem/win32/filesystem_impl.cc:139] read failed with le 32
[3928][warning][misc] [test/test_common/environment.cc:163] move completed E:\temp/eds.update.pb_text -> E:\temp/eds.pb_text
[9068][warning][config] [source/common/config/filesystem_subscription_impl.cc:78] Filesystem config update failure: unable to read file: E:\temp/eds.pb_text

Additional Description: Solved by retrying
Risk Level: Low
Testing: Existing integration tests
Docs Changes:
Release Notes:
Platform Specific Features:

@sunjayBhatia I created as a baseline to see how we deal with this issue.

MoveFileEx should be atomic, but from the second to last post here: https://social.msdn.microsoft.com/Forums/windowsdesktop/en-US/449bb49d-8acc-48dc-a46f-0760ceddbfc3/movefileexmovefilereplaceexisting-ntfs-same-volume-atomic?forum=windowssdk it seems MoveFileEx can revert back to not being atomic even if the underlying filesystem supports it

Thanks for digging this up. So this means that we are catching the notification from Delete and acting on that while the file gets copied.
I am wondering why the Delete is triggering a notification though. In watcher_impl
    uint32_t events = 0;
    if (fni->Action == FILE_ACTION_RENAMED_NEW_NAME) {
      events |= Events::MovedTo;
    }
    if (fni->Action == FILE_ACTION_MODIFIED) {
      events |= Events::Modified;
    }

    constexpr absl::string_view data{"a"};
    for (FileWatch& watch : dir_watch->watches_) {
      if (watch.file_ == file && (watch.events_ & events)) {

I would assume that a delete file you have the following action FILE_ACTION_REMOVED.
My explanation for the race is that MoveFile copy operation is triggering an update in FILE_NOTIFY_CHANGE_LAST_WRITE before the move completes and this is the root cause of the race

Yeah, would be interesting to verify if MoveFileEx is in fact falling back to non-atomic operations, and what they are for a move to a file that already exists, given that the post was likely talking about a move to a file that doesn't yet exist since it mentions copy + delete
We should be only registering interest in move events, so I would hope deletions don't register: https://github.com/envoyproxy/envoy/blob/0cb98ff9a0432748d4485c7dadfd182c077065a7/source/common/config/filesystem_subscription_impl.cc#L22

And we should only be watching modified/file name changes:

FILE_NOTIFY_CHANGE_FILE_NAME is triggered when a file gets deleted. But as you point out we filter them out in the watcher_impl because the subscriber only cares for MoveTo

Yeah, would be interesting to verify if MoveFileEx is in fact falling back to non-atomic operations, and what they are for a move to a file that already exists, given that the post was likely talking about a move to a file that doesn't yet exist since it mentions copy + delete

I could ask internally and figure out the implementation details. However, I am not sure if this would be the best solution for the customer. I am thinking that there could be scenario when someone copies the configuration file (either with MoveFileEx or with some high level language that calls this Win32 api under the hood). In that case the user might encounter this bug. This is why I opted to solve with retry on the production code rather than tinkering with the test code.

Could another approach be to not send the notification to libevent or not call the callback until we are sure the file is unlocked? Retry Open or CreateFile until it succeeds, so we encapsulate the functionality/workaround in the win32 watcher

Repro with debug level logs
[2020-12-15 21:04:37.394][3780][debug][config] [source/common/config/filesystem_subscription_impl.cc:71] Filesystem config update accepted for C:/botcode/w/_tmp/fc540bb82cc03c13d830162ac0749943/eds.pb_text: version_info: "1"
resources {
  [type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment] {
    cluster_name: "cluster_0"
    endpoints {
      lb_endpoints {
        endpoint {
          address {
            socket_address {
              address: "<IP_ADDRESS>"
              port_value: 49222
            }
          }
        }
      }
    }
  }
}
type_url: "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment"
183412668: "envoy.api.v2.DiscoveryResponse"

[2020-12-15 21:04:37.395][3780][debug][connection] [source/common/network/connection_impl.cc:651] [C24] connected
[2020-12-15 21:04:37.395][3780][debug][client] [source/common/http/codec_client.cc:77] [C24] connected
[2020-12-15 21:04:37.405][3780][debug][client] [source/common/http/codec_client.cc:109] [C24] response complete
[2020-12-15 21:04:37.405][3780][debug][hc] [source/common/upstream/health_checker_impl.cc:297] [C24] hc response=200 health_flags=/failed_active_hc
[2020-12-15 21:04:37.406][2632][debug][file] [source/common/filesystem/win32/watcher_impl.cc:187] notification: handle: 0x1c8 action: 1 file: eds.update.pb_text
[2020-12-15 21:04:37.407][4256][debug][file] [source/common/filesystem/win32/watcher_impl.cc:187] notification: handle: 0x2d0 action: 1 file: eds.update.pb_text
[2020-12-15 21:04:37.407][736][debug][file] [source/common/filesystem/win32/watcher_impl.cc:187] notification: handle: 0x2c8 action: 1 file: eds.update.pb_text
[2020-12-15 21:04:37.407][736][debug][file] [source/common/filesystem/win32/watcher_impl.cc:187] notification: handle: 0x2c8 action: 3 file: eds.update.pb_text
[2020-12-15 21:04:37.407][2632][debug][file] [source/common/filesystem/win32/watcher_impl.cc:187] notification: handle: 0x1c8 action: 3 file: eds.update.pb_text
[2020-12-15 21:04:37.407][4256][debug][file] [source/common/filesystem/win32/watcher_impl.cc:187] notification: handle: 0x2d0 action: 3 file: eds.update.pb_text
[2020-12-15 21:04:37.407][2632][debug][file] [source/common/filesystem/win32/watcher_impl.cc:187] notification: handle: 0x1c8 action: 2 file: eds.pb_text
[2020-12-15 21:04:37.407][4256][debug][file] [source/common/filesystem/win32/watcher_impl.cc:187] notification: handle: 0x2d0 action: 2 file: eds.pb_text
[2020-12-15 21:04:37.407][736][debug][file] [source/common/filesystem/win32/watcher_impl.cc:187] notification: handle: 0x2c8 action: 2 file: eds.pb_text
[2020-12-15 21:04:37.407][4256][debug][file] [source/common/filesystem/win32/watcher_impl.cc:187] notification: handle: 0x2d0 action: 4 file: eds.update.pb_text
[2020-12-15 21:04:37.407][4256][debug][file] [source/common/filesystem/win32/watcher_impl.cc:187] notification: handle: 0x2d0 action: 5 file: eds.pb_text
[2020-12-15 21:04:37.407][2632][debug][file] [source/common/filesystem/win32/watcher_impl.cc:187] notification: handle: 0x1c8 action: 4 file: eds.update.pb_text
[2020-12-15 21:04:37.407][736][debug][file] [source/common/filesystem/win32/watcher_impl.cc:187] notification: handle: 0x2c8 action: 4 file: eds.update.pb_text
[2020-12-15 21:04:37.407][736][debug][file] [source/common/filesystem/win32/watcher_impl.cc:187] notification: handle: 0x2c8 action: 5 file: eds.pb_text
[2020-12-15 21:04:37.407][736][debug][file] [source/common/filesystem/win32/watcher_impl.cc:200] matched callback: file: eds.pb_text
[2020-12-15 21:04:37.407][2632][debug][file] [source/common/filesystem/win32/watcher_impl.cc:187] notification: handle: 0x1c8 action: 5 file: eds.pb_text
[2020-12-15 21:04:37.407][3780][debug][file] [source/common/filesystem/win32/watcher_impl.cc:129] executing callback
[2020-12-15 21:04:37.407][3780][debug][config] [source/common/config/filesystem_subscription_impl.cc:61] Filesystem config refresh for C:/botcode/w/_tmp/fc540bb82cc03c13d830162ac0749943/eds.pb_text
[2020-12-15 21:04:37.408][3780][warning][config] [source/common/config/filesystem_subscription_impl.cc:78] Filesystem config update failure: unable to read file: C:/botcode/w/_tmp/fc540bb82cc03c13d830162ac0749943/eds.pb_text
[2020-12-15 21:04:38.399][3780][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1200] membership update for TLS cluster cluster_0 added 0 removed 0
[2020-12-15 21:04:38.399][2252][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1200] membership update for TLS cluster cluster_0 added 0 removed 0
[2020-12-15 21:04:39.381][3780][info][main] [source/server/drain_manager_impl.cc:70] shutting down parent after drain
[2020-12-15 21:04:42.364][3780][debug][main] [source/server/server.cc:197] flushing stats
[2020-12-15 21:04:47.367][3780][debug][main] [source/server/server.cc:197] flushing stats
[2020-12-15 21:04:52.367][3780][debug][main] [source/server/server.cc:197] flushing stats
[2020-12-15 21:04:57.365][3780][debug][main] [source/server/server.cc:197] flushing stats

With any retry logic I guess we'll have to make a judgement about how many retries to do (could it be that 1 is not enough as you have it now?), is there a pause in between? etc.

The way I implemented it is with exponential back-off and sleeps in between

Could another approach be to not send the notification to libevent or not call the callback until we are sure the file is unlocked? Retry Open or CreateFile until it succeeds, so we encapsulate the functionality/workaround in the win32 watcher

The work around needs to exceed the watcher, because it is the responsibility of the caller who knows if they want to read the file or not. So the caller needs to decide if they want to wait or not. This could be passed as a parameter to the watcher but it needs to be configured by the caller.
Also if we schedule it immediately we have time between the callback being called as an additional wait time which would minimize the time we could potentially wait.

The way I implemented it is with exponential back-off and sleeps in between

Ah gotcha, totally skimmed the retry logic

Thanks a lot for the discussion @sunjayBhatia! I think I have fixed it in another way that does not requires retrying just relying on win32 api.
I will close this one and make another PR with the different fix


Fix compilation errors

Current dependencies on/for this PR:

feature/proof-of-stake

PR #521   👈

This stack of pull requests is managed by Graphite.


[Darc-Update] global.json, Version.props and Version.Details.xml

Darc is trying to update these files to the latest versions found in the Product Dependency Store

Working on it...


Breaking Bug: Randomly appends additional } when serializing and writing to textfile

Hi,
I have a basic class with some properties which I use as Configuration files for my Windows Forms App.
I have written a configuration manager which deals with serialization and deserialization
of the configuration files.
In the constructor of this manager, i will load the configuration, then write it back to the harddrive
to update the JSON again. I do this so if my class introduced new properties to have them within the JSON file and default values.
Randomly the app crashes because of invalid format of the JSON object due to an extra closing } at the end of the file.
I haven't managed to  reproduce this problem in a controlled scenario, it is sometimes randomly 10 out of 10 times it may work and then sometimes it fails twice in the row (after fixing the json  and restarting the app).
//The constructor
internal ConfigurationManager()
        {
            IConfiguration temp = new MasterConfig();
            Refresh(ref temp);
            MasterConfig = temp;
        }
public T Read<T>(bool rewriteIfExists=true) where T : class, IConfiguration
        {
            T defaultInstance = Activator.CreateInstance<T>();
            return Read(defaultInstance, rewriteIfExists) as T;
        }

        public bool Write<T>(T instance) where T : IConfiguration
        {
            string filepath = instance.CreateDirectoryIfNotExist();

            var serialized = Serializer.Serialize(instance);

            using (TextWriter writer = new StreamWriter(File.OpenWrite(filepath)))
                writer.Write(serialized);

            return true;
        }

        private void Refresh(ref IConfiguration cfg)
        {
            cfg = (IConfiguration)Read(cfg);
        }
private object Read(IConfiguration instance, bool rewriteIfExists = false)
        {
            var filepath = instance.GetFilePath();
            if (File.Exists(filepath))
            {
                var content = File.ReadAllText(filepath);
                var loadedCfg = (IConfiguration)Serializer.Deserialize(content, instance.GetType());
                var b = loadedCfg as ConfigurationBase;
                b.PropertyChanged += (sender, property) => {
                    ConfigChanged(b);
                };
                //forces to write new properties which havent been written to cfg yet
                if (rewriteIfExists)
                    Write(loadedCfg);

                return loadedCfg;
            }
            else
            {

            }

            if (!Write(instance))
                throw new Exception("Could not write configuration.");

            return instance;
        }

The serializer class which is just a wrapper:
public class JsonSerializer : Serializer
    {
        #region Overrides of Serializer

        public override string Serialize(object obj)
        {
            return Serialize(obj, Format.Indented);
        }
        public  string Serialize(object obj, Format format)
        {
            Formatting form = Formatting.None;
            if(format == Format.Indented)
                form = Formatting.Indented;

            return JsonConvert.SerializeObject(obj, form);
        }
        public override T Deserialize<T>(string json)
        {
            if (typeof (T).IsInterface || typeof (T).IsAbstract)
            {
                throw new ArgumentException("You cannot use an interace nor an abstract class as a type",nameof(T));
                //var obj = Deserialize(json);
               // return obj as T;
            }

            return JsonConvert.DeserializeObject<T>(json);
        }

        public override object Deserialize(string json, Type type=null)
        {
            return type==null ? JsonConvert.DeserializeObject(json) : JsonConvert.DeserializeObject(json, type);
        }

        #endregion
    }

I assume the problem is the refresh function on the Manager call but I am not doing anything async or parallel so i am wondering why the serialization of the class randomly ends up with an additional } at the end for example and other times it just works:
{
"name":"test"
}}

or sometimes also:
{
"name":"test"
}
}

i tried re-running the final application several times in visual studio but there i haven't had any luck reproducing this. This mystery happens randomly on 3 devices.
The actual json string, this one does not change between APP restarts however it gets overwritten on each app start to update potential configuration when new settings have been introduced.

{
  "DatabaseConnectionString": "Data Source=SOURCE;Initial Catalog=NAME;Persist Security Info=True;User ID=USER;Password=NAME",
  "Emails": [
<EMAIL_ADDRESS>  ],
  "ExceptionEmails": [
<EMAIL_ADDRESS>  ],
  "LocalPath": "C:\\",
  "ServerPath": "C:\\",
  "SourcePath": "C:\\",
  "TrendPath": "C:\\",
  "Name": "GeneralAppConfiguration",
  "Created": "2017-11-01T08:41:29.9736683+11:00"
}

You're using File.OpenWrite - https://msdn.microsoft.com/en-us/library/system.io.file.openwrite(v=vs.110).aspx
Bet you $10 you're writing to a file that already exists which is why you have multiple }'s

Yes, in the cases when it happens it does already exist, however, the content did not change and if, it would get longer which means the original content should be overwritten. In the current case the setting class contains the exact same properties & values which should mean it is the exact same JSON string.
So randomly on app start the format gets corrupted, then I manually fix it and then it might be OK for another X amount of app-starts until the format breaks again.
I can write a workaround and clear the content first, reposition and rewrite it or separate the code if the file exists or if it has to be created first.

Just write to the file in such a way that it is recreated. Writing to the start of a file that already exists is a fundamentally bad idea.


External content and triplestore indexing fix

Part of https://github.com/Islandora-CLAW/CLAW/issues/1079
This PR will build Alpaca from source and deploy all the karaf stuff through the Alpaca role to get Fedora triplestore indexing working again with Fedora 5.0.  This PR temporarily disables the Apix role, which I'll put back once Apix is brought into line as well.
This PR also pulls in a new fcrepo role which templates out the config neccessary for external content.  This fixes how Media is indexed in fedora.
Testing Instructions

Pull down this PR
vagrant up
Make a Repository Item, tag it as 'Image', and then add an 'Original File' media to it.
Go check Gemini, you should now see all files in Gemini, regardless if they're in Fedora or not.  You also should not see media at all in Gemini.  Just files, from which the rdf can be found with a HEAD request.

mysql> select drupal_uri, fedora_uri from Gemini;
+---------------------------------------------------------------------------+------------------------------------------------------------------------------------+
| drupal_uri                                                                | fedora_uri                                                                         |
+---------------------------------------------------------------------------+------------------------------------------------------------------------------------+
| http://localhost:8000/taxonomy/term/5?_format=jsonld                      | http://localhost:8080/fcrepo/rest/00/84/25/19/00842519-a0ab-41d1-9a0a-748fe226cf9c |
| http://localhost:8000/taxonomy/term/1?_format=jsonld                      | http://localhost:8080/fcrepo/rest/0e/8f/24/56/0e8f2456-445e-4977-ba27-271124d616c6 |
| http://localhost:8000/taxonomy/term/2?_format=jsonld                      | http://localhost:8080/fcrepo/rest/0f/20/d6/a1/0f20d6a1-4d52-44de-bf73-a7fb7f84051e |
| http://localhost:8000/taxonomy/term/16?_format=jsonld                     | http://localhost:8080/fcrepo/rest/18/69/80/05/18698005-c031-4ed6-bbc8-0ff8b4992b61 |
| http://localhost:8000/taxonomy/term/14?_format=jsonld                     | http://localhost:8080/fcrepo/rest/27/05/5a/cf/27055acf-d5c3-4cdd-9615-a18f5ca82e30 |
| http://localhost:8000/taxonomy/term/6?_format=jsonld                      | http://localhost:8080/fcrepo/rest/2e/34/90/74/2e349074-a1f1-4aaa-beb4-4e0a15feb6b9 |
| http://localhost:8000/taxonomy/term/25?_format=jsonld                     | http://localhost:8080/fcrepo/rest/2e/71/f0/42/2e71f042-51f3-4f05-8b65-ed2be04f0225 |
| http://localhost:8000/taxonomy/term/24?_format=jsonld                     | http://localhost:8080/fcrepo/rest/34/17/9c/96/34179c96-2b41-422b-9a88-a8754a677213 |
| http://localhost:8000/_flysystem/fedora/2019-04/Arrival-Back-Home.jpg     | http://localhost:8080/fcrepo/rest/2019-04/Arrival-Back-Home.jpg                    |
| http://localhost:8000/taxonomy/term/20?_format=jsonld                     | http://localhost:8080/fcrepo/rest/54/d3/f0/6e/54d3f06e-b9ac-4caf-9d2b-e6bc61189529 |
| http://localhost:8000/node/1?_format=jsonld                               | http://localhost:8080/fcrepo/rest/68/4f/ab/a4/684faba4-9cda-4ce1-9bbf-92d0c2cab2fc |
| http://localhost:8000/taxonomy/term/17?_format=jsonld                     | http://localhost:8080/fcrepo/rest/76/63/26/17/76632617-37d4-421e-bb0e-732b42ccdcc4 |
| http://localhost:8000/taxonomy/term/22?_format=jsonld                     | http://localhost:8080/fcrepo/rest/7d/a1/d7/62/7da1d762-b3c4-41a8-879d-06a741b084cd |
| http://localhost:8000/taxonomy/term/9?_format=jsonld                      | http://localhost:8080/fcrepo/rest/8b/30/ba/21/8b30ba21-a2b0-452b-9d1c-d8ce4fb57cca |
| http://localhost:8000/sites/default/files/2019-04/1-Service%20File.jpg    | http://localhost:8080/fcrepo/rest/96/b2/67/23/96b26723-b08e-4c71-b38f-c818a5e6f3f1 |
| http://localhost:8000/sites/default/files/2019-04/1-Thumbnail%20Image.jpg | http://localhost:8080/fcrepo/rest/a2/0d/cc/b2/a20dccb2-a1b3-41da-abce-90132a096213 |
| http://localhost:8000/taxonomy/term/19?_format=jsonld                     | http://localhost:8080/fcrepo/rest/a3/53/e5/0c/a353e50c-24e5-4d1d-8a0c-2921aa98095a |
| http://localhost:8000/taxonomy/term/10?_format=jsonld                     | http://localhost:8080/fcrepo/rest/a7/d8/79/57/a7d87957-69d5-44dc-ac54-80ada17bb137 |
| http://localhost:8000/taxonomy/term/21?_format=jsonld                     | http://localhost:8080/fcrepo/rest/b2/5e/c8/48/b25ec848-5058-4be1-8ef1-857cdb63f0f0 |
| http://localhost:8000/taxonomy/term/7?_format=jsonld                      | http://localhost:8080/fcrepo/rest/be/45/81/1f/be45811f-fe32-4dc6-979e-e2badb04f69c |
| http://localhost:8000/taxonomy/term/15?_format=jsonld                     | http://localhost:8080/fcrepo/rest/cc/45/4d/c0/cc454dc0-3f43-4df9-a89b-024eb0cb8e33 |
| http://localhost:8000/taxonomy/term/4?_format=jsonld                      | http://localhost:8080/fcrepo/rest/d2/6b/95/55/d26b9555-7177-4050-8b46-a0b211a68e88 |
| http://localhost:8000/taxonomy/term/3?_format=jsonld                      | http://localhost:8080/fcrepo/rest/d9/24/5c/78/d9245c78-dc91-4eaa-9704-5c74b9a4e07b |
| http://localhost:8000/taxonomy/term/18?_format=jsonld                     | http://localhost:8080/fcrepo/rest/dc/7e/02/10/dc7e0210-dfe1-4660-85c4-53fe0c75cb18 |
| http://localhost:8000/taxonomy/term/11?_format=jsonld                     | http://localhost:8080/fcrepo/rest/e6/63/1f/61/e6631f61-a56e-4d83-beb1-b9d9d84c3561 |
| http://localhost:8000/taxonomy/term/13?_format=jsonld                     | http://localhost:8080/fcrepo/rest/ee/84/c3/d1/ee84c3d1-462b-4b68-8610-7841d74defad |
| http://localhost:8000/taxonomy/term/12?_format=jsonld                     | http://localhost:8080/fcrepo/rest/f5/90/31/51/f5903151-9324-46b7-8cc2-d1e07d69ec59 |
| http://localhost:8000/taxonomy/term/8?_format=jsonld                      | http://localhost:8080/fcrepo/rest/fd/bc/90/04/fdbc9004-cd94-40d8-a199-606bd67db7e6 |
| http://localhost:8000/taxonomy/term/23?_format=jsonld                     | http://localhost:8080/fcrepo/rest/ff/1b/82/7e/ff1b827e-f153-4939-b860-dbe866b3f464 |
+---------------------------------------------------------------------------+------------------------------------------------------------------------------------+
29 rows in set (0.00 sec)

HEAD the Original File, which is contained within Fedora.  You should get a 200 OK response.  You can get the url for the file's RDF with the rel="describedby" link header.

vagrant@claw:/var/www/html/Crayfish$ curl -I http://localhost:8080/fcrepo/rest/2019-04/Arrival-Back-Home.jpg
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Cache-Control: private
Expires: Thu, 01 Jan 1970 00:00:00 UTC
Set-Cookie: JSESSIONID=89EA5154319CC643EC7D7403EFDCA0AF; Path=/fcrepo/; HttpOnly
Set-Cookie: rememberMe=deleteMe; Path=/fcrepo; Max-Age=0; Expires=Sun, 28-Apr-2019 18:44:19 GMT
ETag: "bd9c7c8b6f4f486501c6eb01e978240c865b3712"
Last-Modified: Mon, 29 Apr 2019 14:15:01 GMT
Accept-Ranges: bytes
Content-Disposition: attachment; filename=""; creation-date="Mon, 29 Apr 2019 14:15:01 GMT"; modification-date="Mon, 29 Apr 2019 14:15:01 GMT"; size=3166278
Link: <http://www.w3.org/ns/ldp#Resource>;rel="type"
Link: <http://www.w3.org/ns/ldp#NonRDFSource>;rel="type"
Link: <http://localhost:8080/fcrepo/rest/2019-04/Arrival-Back-Home.jpg/fcr:metadata>; rel="describedby"
Link: <http://localhost:8080/fcrepo/rest/2019-04/Arrival-Back-Home.jpg>; rel="timegate"
Link: <http://localhost:8080/fcrepo/rest/2019-04/Arrival-Back-Home.jpg>; rel="original"
Link: <http://localhost:8080/fcrepo/rest/2019-04/Arrival-Back-Home.jpg/fcr:versions>; rel="timemap"
Link: <http://mementoweb.org/ns#OriginalResource>; rel="type"
Link: <http://mementoweb.org/ns#TimeGate>; rel="type"
Accept-External-Content-Handling: copy,redirect,proxy
Allow: DELETE,HEAD,GET,PUT,OPTIONS
Link: <http://localhost:8080/fcrepo/rest/2019-04/Arrival-Back-Home.jpg/fcr:acl>; rel="acl"
Content-Type: image/jpeg
Content-Length: 3166278
Date: Mon, 29 Apr 2019 18:44:19 GMT

Get its RDF and it should be populated correctly (before only system metadata from fcrepo was there, and nothing from Drupal was making it over).

vagrant@claw:/var/www/html/Crayfish$ curl http://localhost:8080/fcrepo/rest/2019-04/Arrival-Back-Home.jpg/fcr:metadata
@prefix premis:  <http://www.loc.gov/premis/rdf/v1#> .
@prefix rdf:  <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix fedora:  <http://fedora.info/definitions/v4/repository#> .
@prefix ebucore:  <http://www.ebu.ch/metadata/ontologies/ebucore/ebucore#> .
@prefix ldp:  <http://www.w3.org/ns/ldp#> .
@prefix dcterms:  <http://purl.org/dc/terms/> .
@prefix iana:  <http://www.iana.org/assignments/relation/> .

<http://localhost:8080/fcrepo/rest/2019-04/Arrival-Back-Home.jpg>
        rdf:type                    fedora:NonRdfSourceDescription ;
        rdf:type                    <http://pcdm.org/models#File> ;
        rdf:type                    <http://pcdm.org/use#OriginalFile> ;
        rdf:type                    fedora:Binary ;
        rdf:type                    fedora:Resource ;
        fedora:lastModifiedBy       "bypassAdmin" ;
        <http://schema.org/dateModified>  "2019-04-29T14:15:13+00:00"^^<http://www.w3.org/2001/XMLSchema#dateTime> ;
        <http://schema.org/author>  <http://localhost:8000/user/1?_format=jsonld> ;
        <http://schema.org/sameAs>  "http://localhost:8000/_flysystem/fedora/2019-04/Arrival-Back-Home.jpg" ;
        <http://schema.org/dateCreated>  "2019-04-29T14:14:55+00:00"^^<http://www.w3.org/2001/XMLSchema#dateTime> ;
        ebucore:width               "2200"^^<http://www.w3.org/2001/XMLSchema#int> ;
        premis:hasSize              "3166278"^^<http://www.w3.org/2001/XMLSchema#long> ;
        ebucore:hasMimeType         "image/jpeg" ;
        fedora:createdBy            "bypassAdmin" ;
        fedora:created              "2019-04-29T14:15:01.808Z"^^<http://www.w3.org/2001/XMLSchema#dateTime> ;
        fedora:lastModified         "2019-04-29T14:15:19.664Z"^^<http://www.w3.org/2001/XMLSchema#dateTime> ;
        premis:hasMessageDigest     <urn:sha1:aaa5b588e151eca37aa5f7cc7462702abae2d3b3> ;
        ebucore:height              "1616"^^<http://www.w3.org/2001/XMLSchema#int> ;
        <http://pcdm.org/models#fileOf>  <http://localhost:8000/node/1?_format=jsonld> ;
        rdf:label                   "Arrival"@en ;
        ebucore:filename            "" ;
        dcterms:title               "Arrival"@en ;
        rdf:type                    ldp:NonRDFSource ;
        iana:describedby            <http://localhost:8080/fcrepo/rest/2019-04/Arrival-Back-Home.jpg/fcr:metadata> ;
        fedora:hasFixityService     <http://localhost:8080/fcrepo/rest/2019-04/Arrival-Back-Home.jpg/fcr:fixity> .

Then HEAD a file that's 'external content' in Fedora (either the service file or thumbnail).  You should see a 307 Redirect status code (external content is set up to use Redirect for now):

vagrant@claw:/var/www/html/Crayfish$ curl -I http://localhost:8080/fcrepo/rest/96/b2/67/23/96b26723-b08e-4c71-b38f-c818a5e6f3f1
HTTP/1.1 307 Temporary Redirect
Server: Apache-Coyote/1.1
Cache-Control: private
Expires: Thu, 01 Jan 1970 00:00:00 UTC
Set-Cookie: JSESSIONID=87200D0E58A3168EB00D086E01E139BA; Path=/fcrepo/; HttpOnly
Set-Cookie: rememberMe=deleteMe; Path=/fcrepo; Max-Age=0; Expires=Sun, 28-Apr-2019 18:43:48 GMT
ETag: "b9c2fd2a5e872a2e74ea67daa88f20226d28fa21"
Last-Modified: Mon, 29 Apr 2019 14:15:21 GMT
Accept-Ranges: bytes
Content-Disposition: attachment; filename=""; creation-date="Mon, 29 Apr 2019 14:15:21 GMT"; modification-date="Mon, 29 Apr 2019 14:15:21 GMT"; size=3271893
Link: <http://www.w3.org/ns/ldp#Resource>;rel="type"
Link: <http://www.w3.org/ns/ldp#NonRDFSource>;rel="type"
Link: <http://localhost:8080/fcrepo/rest/96/b2/67/23/96b26723-b08e-4c71-b38f-c818a5e6f3f1/fcr:metadata>; rel="describedby"
Link: <http://localhost:8080/fcrepo/rest/96/b2/67/23/96b26723-b08e-4c71-b38f-c818a5e6f3f1>; rel="timegate"
Link: <http://localhost:8080/fcrepo/rest/96/b2/67/23/96b26723-b08e-4c71-b38f-c818a5e6f3f1>; rel="original"
Link: <http://localhost:8080/fcrepo/rest/96/b2/67/23/96b26723-b08e-4c71-b38f-c818a5e6f3f1/fcr:versions>; rel="timemap"
Link: <http://mementoweb.org/ns#OriginalResource>; rel="type"
Link: <http://mementoweb.org/ns#TimeGate>; rel="type"
Accept-External-Content-Handling: copy,redirect,proxy
Allow: DELETE,HEAD,GET,PUT,OPTIONS
Link: <http://localhost:8080/fcrepo/rest/96/b2/67/23/96b26723-b08e-4c71-b38f-c818a5e6f3f1/fcr:acl>; rel="acl"
Content-Location: http://localhost:8000/sites/default/files/2019-04/1-Service%20File.jpg
Location: http://localhost:8000/sites/default/files/2019-04/1-Service%20File.jpg
Content-Type: image/jpeg
Content-Length: 0
Date: Mon, 29 Apr 2019 18:43:48 GMT

And get its RDF using the rel="describedby" link header

vagrant@claw:/var/www/html/Crayfish$ curl http://localhost:8080/fcrepo/rest/96/b2/67/23/96b26723-b08e-4c71-b38f-c818a5e6f3f1/fcr:metadata
@prefix premis:  <http://www.loc.gov/premis/rdf/v1#> .
@prefix rdf:  <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix fedora:  <http://fedora.info/definitions/v4/repository#> .
@prefix ebucore:  <http://www.ebu.ch/metadata/ontologies/ebucore/ebucore#> .
@prefix ldp:  <http://www.w3.org/ns/ldp#> .
@prefix dcterms:  <http://purl.org/dc/terms/> .
@prefix iana:  <http://www.iana.org/assignments/relation/> .

<http://localhost:8080/fcrepo/rest/96/b2/67/23/96b26723-b08e-4c71-b38f-c818a5e6f3f1>
        rdf:type                    fedora:NonRdfSourceDescription ;
        rdf:type                    <http://pcdm.org/models#File> ;
        rdf:type                    <http://pcdm.org/use#ServiceFile> ;
        rdf:type                    fedora:Binary ;
        rdf:type                    fedora:Resource ;
        fedora:lastModifiedBy       "bypassAdmin" ;
        <http://schema.org/dateModified>  "2019-04-29T14:15:20+00:00"^^<http://www.w3.org/2001/XMLSchema#dateTime> ;
        <http://schema.org/author>  <http://localhost:8000/user/1?_format=jsonld> ;
        <http://schema.org/sameAs>  "http://localhost:8000/sites/default/files/2019-04/1-Service%20File.jpg" ;
        <http://schema.org/dateCreated>  "2019-04-29T14:15:20+00:00"^^<http://www.w3.org/2001/XMLSchema#dateTime> ;
        ebucore:width               "2200"^^<http://www.w3.org/2001/XMLSchema#int> ;
        premis:hasSize              "3271893"^^<http://www.w3.org/2001/XMLSchema#long> ;
        ebucore:hasMimeType         "image/jpeg" ;
        fedora:createdBy            "bypassAdmin" ;
        fedora:created              "2019-04-29T14:15:21.69Z"^^<http://www.w3.org/2001/XMLSchema#dateTime> ;
        fedora:lastModified         "2019-04-29T14:15:23.344Z"^^<http://www.w3.org/2001/XMLSchema#dateTime> ;
        ebucore:height              "1616"^^<http://www.w3.org/2001/XMLSchema#int> ;
        <http://pcdm.org/models#fileOf>  <http://localhost:8000/node/1?_format=jsonld> ;
        rdf:label                   "1-Service File.jpg"@en ;
        ebucore:filename            "" ;
        dcterms:title               "1-Service File.jpg"@en ;
        rdf:type                    ldp:NonRDFSource ;
        iana:describedby            <http://localhost:8080/fcrepo/rest/96/b2/67/23/96b26723-b08e-4c71-b38f-c818a5e6f3f1/fcr:metadata> ;
        fedora:hasFixityService     <http://localhost:8080/fcrepo/rest/96/b2/67/23/96b26723-b08e-4c71-b38f-c818a5e6f3f1/fcr:fixity> .

And it should also look like it's got all the info from Drupal!
Then go check the triplestore.  You should have all those triples from BOTH Fedora and Drupal in there:
SELECT ?s ?p ?o WHERE { <http://localhost:8080/fcrepo/rest/2019-04/Arrival-Back-Home.jpg> ?p ?o }

AND
SELECT ?s ?p ?o WHERE { <http://localhost:8000/media/1?_format=jsonld> ?p ?o }

should return mostly the same triples, with Fedora having a few extra indexed than Drupal because of extra Fedora specific metadata.

@dannylamb
How exactly do we test this?

@Natkeeran I've got to make some updates for the fcrepo role, and then I'll post up how I'm testing it once I can confirm that this still works.

@Natkeeran Updated test instructions in the initial message

OK, so after review, we'll have to slice new versions of Crayfish and Alpaca, then bump the versions in their roles.  Then update requirements.yml to point to those official roles instead of branches.  Then I can clear out the stuff to build from source for Alpaca.
Then once the islandora and islandora demo PRs are meregd I can clear out some composer.json shenanigans in islandora_demo.
Then this can get merged :tada:

I started this PR up ~22 minutes ago and it just failed on tomcat:
TASK [Islandora-Devops.tomcat8 : template /opt/tomcat/bin/setenv.sh] ***********
Monday 29 April 2019  13:24:25 -0700 (0:00:00.040)       0:21:54.887 **********
fatal: [default]: FAILED! => {"changed": false, "msg": "AnsibleUndefinedVariable: 'fcrepo_allowed_external_content_file' is undefined"}

Ah, did you blow up roles/external? I should probably add that to the
testing instructions.
On Mon, Apr 29, 2019, 17:25 Seth Shaw<EMAIL_ADDRESS>wrote:

I started this PR up ~22 minutes ago and it just failed on tomcat:
TASK [Islandora-Devops.tomcat8 : template /opt/tomcat/bin/setenv.sh] ***********
Monday 29 April 2019  13:24:25 -0700 (0:00:00.040)       0:21:54.887 **********
fatal: [default]: FAILED! => {"changed": false, "msg": "AnsibleUndefinedVariable: 'fcrepo_allowed_external_content_file' is undefined"}
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Islandora-Devops/claw-playbook/pull/100#issuecomment-487730407,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE6PSH644YHCQSC7KGNJB3LPS5KUJANCNFSM4HGJLQKQ
.

@dannylamb, I realized that I hadn't wiped roles/external afterward. Anyway, I have a new problem, this time with adding Alpaca features (specifically fcrepo-service-activemq and fcrepo-indexing-triplestore):
TASK [Islandora-Devops.alpaca : Add Alpaca Features] ***************************
Monday 29 April 2019  16:17:40 -0700 (0:00:00.048)       0:30:38.783 **********
failed: [default] (item=fcrepo-service-activemq) => {"changed": false, "item": "fcrepo-service-activemq", "msg": "client: Ignoring predefined value for KARAF_HOME\nError executing command: Unable to resolve root: missing requirement [root] osgi.identity; osgi.identity=fcrepo-service-activemq; type=karaf.feature; version=\"[5.0.0,5.0.0]\"; filter:=\"(&(osgi.identity=fcrepo-service-activemq)(type=karaf.feature)(version>=5.0.0)(version<=5.0.0))\" [caused by: Unable to resolve fcrepo-service-activemq/5.0.0: missing requirement [fcrepo-service-activemq/5.0.0] osgi.identity; osgi.identity=activemq-camel; type=karaf.feature; version=\"[5.14.0,6.0.0)\"]\n"}
changed: [default] => (item=fcrepo-camel)
changed: [default] => (item=fcrepo-service-camel)
failed: [default] (item=fcrepo-indexing-triplestore) => {"changed": false, "item": "fcrepo-indexing-triplestore", "msg": "client: Ignoring predefined value for KARAF_HOME\nError executing command: Unable to resolve root: missing requirement [root] osgi.identity; osgi.identity=fcrepo-indexing-triplestore; type=karaf.feature; version=\"[5.0.0,5.0.0]\"; filter:=\"(&(osgi.identity=fcrepo-indexing-triplestore)(type=karaf.feature)(version>=5.0.0)(version<=5.0.0))\" [caused by: Unable to resolve fcrepo-indexing-triplestore/5.0.0: missing requirement [fcrepo-indexing-triplestore/5.0.0] osgi.identity; osgi.identity=org.fcrepo.camel.fcrepo-indexing-triplestore-blueprint; type=osgi.bundle; version=\"[5.0.0,5.0.0]\"; resolution:=mandatory [caused by: Unable to resolve org.fcrepo.camel.fcrepo-indexing-triplestore-blueprint/5.0.0: missing requirement [org.fcrepo.camel.fcrepo-indexing-triplestore-blueprint/5.0.0] osgi.service; effective:=active; filter:=\"(&(objectClass=org.apache.camel.Component)(osgi.jndi.service.name=fcrepo/Broker))\"]]\n"}
changed: [default] => (item=islandora-http-client)
changed: [default] => (item=islandora-indexing-triplestore)
changed: [default] => (item=islandora-indexing-fcrepo)
changed: [default] => (item=islandora-connector-derivative)

I have had some different errors, but the last build produced the same as @seth-shaw-unlv

I had some missing things I forgot to push up that were cached in my roles/external.  This should work now.

@seth-shaw-unlv OK, one last vagrant up to test.  I think we've merged everything in, sliced all appropriate versions, and updated all references in the playbook.  Here's hoping :wave: :game_die: :

...almost forgot to wipe roles/external!

Build succeeds, media indexed as expected. Nice work everyone!


fix some bugs/issues of the training scripts

Thanks!  Merging.

The scripts can now fully run with latest k2 and lhotse, but still have some issues:

https://github.com/lhotse-speech/lhotse/issues/125 causes we cannot run many batches successfully.
The modle output is diverging, may because learning rate or model struct, we definitely need to look into this.
Sometimes we may get Cuda out of memory, but because 1, we can not repro it.


yidas/yii2-bower-asset This won't be necessary

    'aliases' => [
        '@bower' => '@vendor/yidas/yii2-bower-asset/bower',
    ]

/dist/js/bootstrap.bundle.js not fond

The wiki outlines the most recent configuration: https://github.com/charlesportwoodii/yii2-psr7-bridge/wiki/RoadRunner-2.9--|-Yii2-Configuration-with-yii-app-basic as of April 2022.

    'aliases' => [
        '@bower' => '@vendor/yidas/yii2-bower-asset/bower',
    ]

/dist/js/bootstrap.bundle.js not fond

        "php": ">=7.4.0",
        "yiisoft/yii2": "~2.0.45",
        "yiisoft/yii2-bootstrap5": "~2.0.2",
        "yiisoft/yii2-symfonymailer": "~2.0.3",

They've upgraded.  There's no upgrade on your end

The dependencies listed in the wiki are the minimum tested. You are not restricted to the minimums if newer ones work.


Adding support for iw in addition to wireless_tools

Is it possible to support iw in addition to wireless_tools?
wireless_tools has been deprecated, as stated in the official Linux Wireless wiki.
Presently, attempting to use the network module with only iw installed results in the following error:
error: Disabling module "wlan" (reason: No built-in support for 'internal/network')
Would be really nice if we could use the network module without requiring wireless_tools.

See #1009.


doc: fix misspellings in docs

Scan and fix misspellings missed during regular reviews.
Signed-off-by: David B. Kinder<EMAIL_ADDRESS>

Codecov Report

Merging #8080 into master will increase coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #8080      +/-   ##
==========================================
+ Coverage   64.55%   64.55%   +<.01%
==========================================
  Files         420      420
  Lines       40141    40141
  Branches     6763     6763
==========================================
+ Hits        25913    25914       +1
+ Misses      11106    11105       -1
  Partials     3122     3122

Impacted Files
Coverage Δ

samples/philosophers/src/main.c
98.43% <0%> (+1.56%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 43553da...29449f4. Read the comment docs.


NIFI-13141: Adding a dragging flag to not reset positioning for labels and connection bends if the user is currently performing an action

NIFI-13141:

Adding a dragging flag to not reset positioning for labels and connection bends if the user is currently performing an action.

reviewing...


Make parsers part of MLBlock

Since each JSON file being parsed represents an MLBlock, the parsing methods should be part of the MLBlock itself, or at least called from within its code.

This will be fixed in #43


Add docker-compose start alias

@robbyrussell How long does it take to accept a pr? What can I do to accelerate it? This one looks like a very simple pr.


Extend set of WMMA operator shapes

Failing tests seem to be configurations that are expected failures, e.g. compute_warp (16, 32, 16) and compute_op_shape. These should throw ConfigErrors after #180


createAzureFunctionsMiddleware mentioned in README.md does not exist in code

Hi, I believe there is a mismatch with the code and the README.md. (JavaScript is not my first language, however :) )
I am not sure which you prefer, but following the README I had to replace createAzureFunctionsMiddleware with createAzureFunction.

You are correct, the README needs to be updated, createAzureFunctionsMiddleware is wrong, the correct function name is createAzureFunction. Thank you!
Would you like to send a pull request yourself?

You are correct, the README needs to be updated, createAzureFunctionsMiddleware is wrong, the correct function name is createAzureFunction. Thank you!
Would you like to send a pull request yourself?

You are correct, the README needs to be updated, createAzureFunctionsMiddleware is wrong, the correct function name is createAzureFunction. Thank you!
Would you like to send a pull request yourself?

Sure, I posted a PR!


Please why are you removing my issue without any reason?

Is there a parameter that corresponds to minimum samples per leaf for the random forest?

min_child_weight is the minimum samples in regression tasks. As for classification tasks, there is no such parameter.


Fix a few typos in comments

Coverage remained the same at 95.29% when pulling 4aa0441b42c67ac03749ba43ef9e894e1b89a934 on nuss-justin:patch-1 into 6f8a5734602782ce3ab3874b474a1b215acc7ed8 on nats-io:master.


Eqref (and hyperref) handling

it should be doable to keep track of \label{...} and keep some form of count then restitute the whole thing when a \eqref or \ref (\\ref{([^{}]*)}) is seen.
In html processing, should use an anchor (see tutorial)
Actions

implement something that processes \command{...} (see #5 also)
have another dictionary (not JD_LOC_VARS because you don't need to keep track of types) to keep track of names->number pair.
might want to keep macro variable such as section variables that are incremented every time a # or ## etc is seen. could also use that for section numbering (if required)

need a  (anchor)
need a Click here to jump

eqref is now handled, it could form the basis for an idea for figures and tables when looking into that.


Common interfaces between RDD, DStream, and DataFrame

This PR is the beginning of an attempt to put a common interface between the main distributed collection types in Spark.
I've tried to make this first checkin towards such a common interface as simple as possible.  To this end, I've taken RDDApi from the sql project, pulled it up into core (as RDDLike), and changed it as necessary to allow all three main distributed collection classes to implement it.
I've then done something similar for pair methods, between RDD and DStream (I don't think there is an equivalent for DataFrames)
This involves a few small interface changes - things like reduceByKey having different method signatures in different classes - but they are, for the moment, minor.  That being said, they are still interface changes, and I don't expect this to get merged in without discussion.  So - suggestions and help are welcome, encouraged, etc.
In the very near future, if this PR is accepted, I would like to expand on it in a few simple ways:

I want to try to pull more functions up into this interface

There are a lot of functions with 3 versions:

foo(...)
foo(..., numPartitions: Int)
foo(..., partitioner: Partitioner)

These should all be replaceble by

foo(..., partitioner: Partitioner = defaultPartitioner)

with the implicit Int => Partitioner conversion I've put in here.  I did half of this reduction, in once case (reduceByKey) out of necessity, trying to get the implementation contained herein to compile, but extending it as far as possible would make a lot of things much cleaner.

Test build #30503 has started for   PR 5565 at commit fb920ff.

Test build #30503 has finished for   PR 5565 at commit fb920ff.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30503/
Test FAILed.

This is such a significant change that it certainly should have a JIRA, and should probably start with design discussion first. This can accompany it as a straw-man, sure. But before even that happens --
My initial reaction is that this is introducing a lot of change to the APIs, which isn't backwards compatible. That's a non-starter for the short term of course, not necessarily in the longer-term, but to core methods, it would still have to pull its weight.
Can these changes be made without changing the API though?
Unifying RDD and DStream may not be important. They expose similar-ish APIs but are different things, and as I understand DStream's methods are partly redundant at this point anyway. I don't have as good a view into how interchangeable an RDD and DataFrame is intended to be.
How much does this much change buy -- what's the upside?

Thanks for taking a look at it.
I'll take a look and write up here exactly what API changes this makes - while I don't think there are many, there aren't none, and that's why I was looking for comments before going any further, once I figured out if it were possible.
As far as I could tell, I could not get this to compile without making these changes, for reasons I'll comment on individually when I get together a list.
As far as the importance of unifying RDD and DStream - I would put that far above unifying RDD and DataFrame.  I only included DataFrame because someone had already made it inherit from a class they called RDDApi, which looked to me like a little bit of prep work for doing what I've done.  RDD and DStream, while different things, they are used for the same purpose.  One of the chief benefits of Spark, touted from early on, was that one could use the same code in varying circumstances - batch jobs, live jobs, streaming jobs, etc.  Yet, from the beginning of the streaming side, despite this being touted as a benefit, it wasn't really true.  I think making it really true is a huge up-side.  I know, for my company, the ability to take our batch jobs and apply them to streaming data without changing our code would be huge, and I can't imagine this isn't true for many other people.

Ok, I think there are only 3 real API changes, though one of them appears in multiple places.

RDD

flatMap[U](f: T => TraversableOnce[U]) changed to flatMap[U](f: T => Traversable[U]).  My thinking at the time was, DStream used a function to Traversable, RDD to TraversableOnce, and that the Traversable part in DStream was necessary there.  Thinking about it again, I'm not completely sure of that - I think it might be just as valid to change the DStream version to TraversableOnce, which would probably be more general, and require fewer people to change code

PairRDDFunctions

Several of the functions now require a ClassTag, as the DStream version already did.  I tried to take the requirement out of the DStream versions instead, since that would be more inclusive, but ran into some compiler problems, and this was easier in the short run.  If people think it's doable to change in that direction, I can try.

combineByKey
mapValues
flatMapValues

reduceByKey

changed the (partitioner, function) version to (function, partitioner) (see below).  I wanted to leave the old version in, deprecated, but for the same reason as the next point, couldn't.
eliminated the (function, int) version.  Unfortunately, the compiler seemed to want to pick the type of function generification before it picked which version of the function to use, and because of this, couldn't handle the type inference.  I'm not sure someone with more scala foo couldn't solve this.  I'll copy the error in here when I can get hold of it again.

However, regarding the last two, there is a general pattern in a lot of the spark functions, that they all have three versions:

foo(....) - an N-argument version, that uses the default partitioner
foo(..., Int) - an N+1-argument version, that uses a HashPartitioner, and
foo(..., Partitioner) - an N+1 argument version, that uses the passed-in partitioner.

PairRDDFunctions.reduceByKey broke this pattern, but PairDStreamFunctions.reduceByKey did not - so, with differing method signatures, one had to change.  Since the RDD one was out of sync with pretty much every similar function in the class, it seemed natural to change it - and, I would argue, only makes sense that way.
As my initial PR notes mention, I think the ideal thing would be to reduce all these function triplets to a single version, with auto-conversion from Int to HashPartitioner, which would leave all associated classes cleaner, smaller, and less confusing.
Besides these three changes, I don't see any other API changes, but I'll go through it again later to be sure.

@nkronenfeld the thrust of my comment comes from the fact that DStreams give you an RDD, which you can do whatever you want with, including make a DStream again with the result via transform. Is that the missing piece here? This change is certainly not a prerequisite for reusing code across batch and streaming.

No, we do that at the moment.
But doing it that way results in a few rather ugly constructs in the application code that can be rather painful, as soon as one starts passing data constructs around.  As soon as one starts passing the collection structures between modules, say, for instance, between stages in a pipeline, one instantly needs to duplicate the entire pipeline for batch and streaming cases.
It isn't just one place where one has to do this replacement - it's every little pipeline operation, for every algorithm, 90% of which are using just the most basic RDD and DStream functions should be easily consolidated.
I'd also note that, where there is an interface change, it is there because the original methods in RDD and DStream were declared inconsistently.  Unless there is a good reason to keep them inconsistent (which so far I don't see in any of these three cases), I would suggest that isn't a good thing to begin with - just in terms of consistency and usability of the library, where they can be the same, they should be.  It reduces the learning curve, and removes some esoteric, hard-to-track-down gotchas that are bound occasionally to bite people newly switching from one case to the other.
On a final note, if this is the intended use of dstream, why have the map, flatMap, reduceByKey, etc functions on it at all?  It seems clear it was intended to be used this way (Hm, that reminds me of a fourth small interface change I'll add above, but as you'll see, it's very, very minor), so why not make sure the use is the same?

Hey @nkronenfeld - simply put, we cannot make binary-incompatible changes to API's in Spark due to our API guarantees, so this rules out many of your proposed changes. This has been true since Spark 1.0. Basically code compiled against older versions needs to work with newer ones, and that sometimes means swallowing minor API inconsistencies.
If you'd like to add new versions of functions with more arguments, or add shared parent interfaces in a way that is binary compatible, that's not a violation of the policy. In those cases though, a JIRA might be helpful to sort of summarize your high level thoughts.

What has to be written differently for an RDD from a DStream vs an RDD from a batch process? they're both RDDs. Maybe a very small example?
I think methods like DStream.map are mostly redundant, yes. I seem to recall @tdas saying something to that effect. I imagine they might not have even been added if this were done over again.
What's different about the API though? the ClassTags? I only skimmed your change and that's all I saw.
But that makes many of the DStream methods at worst superfluous and doesn't change your ability to access the data through the same RDD API.

@pwendell - something like reduceByKey having the opposite argument order of everything else in the code seems like an obvious mistake in the API - is there no mechanism to allow for fixing such things?

@srowen - the examples are pretty simple - it happens as soon as you are passing an RDD to something, rather than passing something to an RDD.  For instance:
case class PipelineData (config: Properties, data: RDD);

In each individual case, there are ways around it, of course, but in total they get to be rather ugly rather quickly.

Test build #30589 has started for   PR 5565 at commit 07c2a13.

Test build #30589 has finished for   PR 5565 at commit 07c2a13.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30589/
Test FAILed.

@nkronenfeld that's a good example. reduceByKey with a Partitioner has it as the first arg in RDD but second in DStream. I'm almost sure that was not intentional. That could be fixed by deprecating the existing method and introducing the corrected one. If it's purely for consistency, I suppose I'd shrug at it, whether it's useful.
You're saying it's for a much more important purpose, which is being able to use the same code consistently between batch and streaming. However I am still missing why in your example you can't apply the same PipelineData to an RDD from DStream.foreachRDD, for example, as to any other RDD?

@nkronenfeld we can do clean-ups in a way that's backwards compatible. But we cannot remove old method signatures for the purpose of clean-up. This is why we vet new public API's pretty closely in Spark post 1.0. However, there is some legacy stuff that, to your point, is sometimes messy. The reason is that otherwise you break old programs that depend on the existing API's.

@pwendell I get all that... does that mean we are stuck with those deprecated versions forever then? Or is there some timeframe over which deprecated functions will eventually be removed?  Would it be possible to prepare for this, say, 4 releases in advance, so eventually it could be done?

Yes, you would deprecate rather than remove methods and then likely remove them in the next major release, like a 2.0.0 release. That's a straightforward process, but first the question is whether it is worth doing, and it's probably not purely for consistency. That's why the question is still whether there is something this is buying that you can't accomplish with the RDD API itself in both batch and streaming.

@srowen are you suggesting putting the entire application inside the foreachRDD call?

@nkronenfeld I suppose this argument does depend on whether I'm right about this: you could express anything you can express with a bunch of DStream operations ending in a bunch of actions, using one big call to foreachRDD. Where does this most conflict with your experience with what you're trying to do? that might suss out the problem here one way or the other rapidly.

I think the DStream API does expose calls which basically call foreachRDD, like the map etc. But from an app developer point of view, it makes my application much cleaner if I do something like {{dstream.map(_.getValue)}} than having a foreachRDD call for every transformation. I don't even see a major maintenance issue as far as the codebase is concerned, since the DStream API mostly just use public API. Really, removing these methods will just make streaming apps more complex.

@harishreedharan Yeah I'm saying the DStream methods are convenience methods. If that's right, you could argue they should be removed, but I'm not arguing that. The question is more like, do they have to be formally unified with the RDD API to enable operating across batch and streaming?

@srowen - If I understand your question, you're asking why we can't take our application, written for an RDD, and just put the whole thing inside a call to dstream.foreachRDD... I'm not ignoring it, I'm trying to figure out what would happen if we did that, and that might take a bit.
Regarding your answer to @harishreedharan, I would say, of course we don't have to - but we really should have from the start.  To have two nearly, but not quite, identical APIs is confusing, and encourages inconsistency; having a common interface enforces that the methods in both mean the same thing now, change together in the future, and are consistent.

Sure, I hope it's good news, that the theory and promise matches reality -- that you can do just about anything you can do in Streaming by manipulating an RDD and that manipulating an RDD is the same no matter whether you're in streaming or not.
I think @tdas would also agree it should have been consistent -- seems like just a little oversight -- and maybe DStream didn't need this similar API at all. I agree with that. What can or should be done going forward is a different question. I tend to not use the DStream methods at all where I can manipulate an RDD, myself, so it doesn't create much of an issue in practice from my POV.

I agree that they should have been unified from the beginning, but at this point merging the two APIs into one would have to be done when we can break version compat. I'd still like to have something like DStream.map (in fact, DStream could really have been an RDD implementation itself, so we didn't have multiple implementations).
At this point, as long as we can unify APIs without breaking them I am ok with it - and in a future 2.0 release, we can make DStream an RDD implementation.

@srowen I'm sure, for what it's worth, that part works out - it's whether the surrounding application structure can fit in that paradigm that is the issue.
Just to be clear, I didn't really expect this to ever get merged in as is :-)  I am hoping for a few things from this PR:

I wanted a list of ununified points the various APIs needed to be unified
I wanted to know what sort of compatibility was needed (code-compatible, which with one exception, this is, vs. binary compatible, which it definitely isn't)
I was hoping to foster in the community some sense that this was a goal we could work towards

So that when we got to the next point where we could make compatibility changes, we would be ready to do so
And to try and prevent further changes from moving the APIs even farther appart.

And, lastly, I was hoping that someone whos scala was better than mine might have some ideas on how to do this without as many API changes (for instance, modifying this so that we could have all 4 versions of reduceByKey, with the bad one deprecated, and still have the compiler intuit types correctly)

Yes, good to see a list of the differences. I suspect that many of these changes (removing DStream methods? at least making them consistent?) could easily happen in 2.x. You can file a Improvement JIRA targeted for 2+ as a TODO; that's better than a PR in the long term, since this should be resolved one way or the other in the short term.
If it's not going to be committed do you mind closing it? it will still be here to inspect.
Adding methods in a backwards-compatible way mostly amounts to keeping and deprecating existing methods. It's possible where needed. I think it's not worth bothering with that at this stage, given that the methods in question are arguably superfluous. That's not a claim that it couldn't be migrated, or that the existing API is ideal.

Could you leave it open a little bit longer? I want to see if I can shorten the list of differences, and having it active here makes it a little easier to see if I've done it correctly.

OK, mark it [WIP] in the title, but does it help you much to have it as an open PR? you can always test things locally.

I'll see, I guess I can always reopen.
If I can get some work towards this done, some deprecation with new versions of the necessary functions, I'll resubmit.
Any idea how to deal with the classtag differences?


Add MapEntry formatting based on Representation.toStringOf().

Add special  processing of MapEntry for pretty printing of AssertionError message.
   /* Pull request solves this issue */
  @Test public void mapEntryIssue() {
        assertThat(Collections.singletonMap("one-two-three", new long[] {1, 2, 3}))
            .containsEntry("one-two-three", new long[] {3, 2, 1});
        /* generates
           java.lang.AssertionError:
                Expecting:
                 <{"one-two-three"=[1L, 2L, 3L]}>
                to contain:
                 <[MapEntry[key='one-two-three', value=[J@57a4d5ee]]>
                but could not find:
                 <[MapEntry[key='one-two-three', value=[J@57a4d5ee]]>
         */
    }

Changes update some tests and break convention on encoding strings in MapEntry with a single quote - instead StandardRepresentation is used.

Good idea, thanks !

Integrated, thanks !


develop the MPI/GPU part

This part of the workshop on MPI and GPUs with apptainer needs more work to develop and test- I suspect pretty much all of it will change, but I'm happy to be wrong.

Not planning to do this for the June workshop- requires creating containers for Myriad. Which is a worthy project, but not in scope for me.

closing


StyleBoxTexture axis stretch tile and tilefit doesn't work when using GLES2

Godot version:
v3.1.2
OS/device including version:
Windows 10 pro education 64 bit
Issue description:
styles that use tile or tilefit as their axis stretch just stretch it and don't actully tile it.

Steps to reproduce:

make new theme
create the hscrollbar templates
make the hscrollbar scroll style use a tile streach mode for both axes
back out to theme editor and see it still stretches and doesn't tile at all

Minimal reproduction project:
GUITest.zip

3.2.2 stable, still a thing.

Can still reproduce on v3.5.2.stable.official [170ba337a]

Duplicate of https://github.com/godotengine/godot/issues/26168.


HDDS-4556. Add smoketest for ozonescripts environment

What changes were proposed in this pull request?
start-ozone.sh and stop-ozone.sh are currently not exercised by any tests, so they can be inadvertently broken without CI feedback.  This PR adds a smoketest for ozonescripts environment, which uses these two scripts.
https://issues.apache.org/jira/browse/HDDS-4556
How was this patch tested?
https://github.com/adoroszlai/hadoop-ozone/runs/1512936707

Thanks @elek for reviewing and committing it.

Thanks @elek for reviewing and committing it.


Help

This issue was unintentionally created by me, so it is meaningless. If the administrator sees this issue, please delete it promptly. Thank you very much.

This issue was unintentionally created by me, so it is meaningless. If the administrator sees this issue, please delete it promptly. Thank you very much.


Adding native ads in Flutter

I would love to ad native ads into my code in Flutter but I can't deal with it. I'm using this: https://pub.dev/packages/firebase_admob
They say:
Native Ads are presented to users via UI components that are native to the platform. (e.g. A View on Android or a UIView on iOS). Using Flutter widgets to create native ads is NOT supported by this.
Since Native Ads require UI components native to a platform, this feature requires additional setup for Android and iOS
package my.app.path;

import com.google.android.gms.ads.formats.UnifiedNativeAd;
import com.google.android.gms.ads.formats.UnifiedNativeAdView;
import io.flutter.plugins.firebaseadmob.FirebaseAdMobPlugin.NativeAdFactory;
import java.util.Map;

class NativeAdFactoryExample implements NativeAdFactory {
  @Override
  public UnifiedNativeAdView createNativeAd(
      UnifiedNativeAd nativeAd, Map<String, Object> customOptions) {
    // Create UnifiedNativeAdView
  }
}

But I can't figure it out how to create "// Create UnifiedNativeAdView"

/* AppDelegate.m */

#import "FLTFirebaseAdMobPlugin.h"

@interface NativeAdFactoryExample : NSObject<FLTNativeAdFactory>
@end

@implementation NativeAdFactoryExample
- (GADUnifiedNativeAdView *)createNativeAd:(GADUnifiedNativeAd *)nativeAd
                             customOptions:(NSDictionary *)customOptions {
  // Create GADUnifiedNativeAdView
}
@end

Could you please give me a solution for Android and IOS?
And how it can be put into the Column or ListVIew in dart?

Hi @JakubKlementewicz
It seems you are asking for assistance on personal code. Please see https://flutter.dev/community for resources and asking questions like this,
you may also get some help if you post it on Stack Overflow and if you need help with your code, please see https://www.reddit.com/r/flutterhelp/
Closing, as this isn't an issue with Flutter itself. If you disagree, please write in the comments and I will reopen it.
Thank you


Propagate UTxO-HD to ouroboros-consensus-test

Description

description of the pull request, if it fixes a particular issue it should link
the PR to a particular issue, see
ref
Checklist

Branch

[ ] Commit sequence broadly makes sense
[ ] Commits have useful messages
[ ] The documentation has been properly updated
[ ] New tests are added if needed and existing tests are updated
[ ] Any changes affecting Consensus packages must have an entry in the appropriate changelog.d directory created using scriv. If in doubt, see the Consensus release process.
[ ] If this branch changes Network and has any consequences for downstream repositories or end users, said changes must be documented in interface-CHANGELOG.md
[ ] If serialization changes, user-facing consequences (e.g. replay from genesis) are confirmed to be intentional.

Pull Request

[ ] Self-reviewed the diff
[ ] Useful pull request description at least containing the following information:

What does this PR change?
Why these changes were needed?
How does this affect downstream repositories and/or end-users?
Which ticket does this PR close (if any)? If it does, is it linked?

[ ] Reviewer requested

Current dependencies on/for this PR:

master

PR #4352

PR #4341

PR #4342

PR #4343

PR #4344

PR #4363   👈

PR #4316

This comment was auto-generated by Graphite.


How do I change the sourceMappingURL?

I've tried all I can think of in order to change the printed URL in the script file:
//# sourceMappingURL=my-module.js.map
But no matter what I've tried the path never changes. Firefox won't load my map until I manually change it to (adding 'assets/js/' manually):
//# sourceMappingURL=assets/js/my-module.js.map
So, my question is: Which one of the more or less obscure options do I have to change to make this work?
Now, the maps get created just fine in the right location (next the the script file), but the printed map path never changes.

.pipe(sourcemaps.write('assets/js')) should be the solution.


Insecure connection could allow RCE

If the server is installed on Termux, then it seems as though anyone could, knowing your IP address, easily connect to the websocket unauthenticated and have RCE through Termux, which is especially bad if your phone is rooted.

Yes, but this is local server no one can access other than you, but if you will give your phone then  it may be insecure.
Or if you have any suggestions for this , you can provide. i will happy to see.

If we use the server and also surfing the web...
Then a evil (?) page surely knows your one of your IP as <IP_ADDRESS>, and he cound connect to it easily by start a WebSocket and grab a shell of your phone...
Port could be scanned, too.

Ok , In next update I will add authentication system ♥♥